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ABSTRACT 

The theme of the 2001 annual conference of the Northeast 
Association for Institutional Research was Institutional Research: Leadership 
through Excellence. These proceedings represent the intellectual content and 
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Nugent) ; (4) Organizational and Technical Issues in Implementing Web-Based 

Surveys: A Case Study (Gordon Hewitt, Dawn Geronimo Terkla, Heather S. 

Roscoe, and Jennifer Dyer) ; (5) Comprehension of Student Success among 

Asnuntuck Community College Elementary Algebra Students Placed by Accuplacer 
Score, Scholastic Aptitude Test Score, or Prerequisite Course (Patricia 
Hirschy and Qing Mack) ; (6) Assessing the Economic Impact of a University on 

the State and Community (Heather Kelly Isaacs) ; (7) Institutional Research 

and Effectiveness: Where Research Meets Strategic Planning (Ana Lucia 
Kazan- Fishman) ; (8) The Impact of a Series of Writing- Intensive Courses on 

Success on the Writing Proficiency Requirement (Kevin B. Murphy) ; (9) An 

Analysis of Market Factors, Fiscal Strength, and Institutional 
Competitiveness (Erika M. Newcomer and Jason P. Casey) ; (10) The Impact of 
Instructional Delivery on Learning Outcomes and Items To Persist (Kristin A. 
Owens and J. Fredericks Volkwein) ; (11) The Impact of Lottery Incentives on 

Student Survey Response Rates (Stephen R. Porter and Michael E. Whitcomb); 

(12) Administering Surveys on the Web: Methodological Issues (Heather S. 
Roscoe, Dawn Geronimo Terkla, Gordon J. Hewitt, and Jennifer A. Dyer) ; (13) 

Student Outcomes: The Impact of Varying Living-Learning Community Models 
(Martha L. A. Stassen) ; (14) Student Satisfaction: Measures and Measurements 

(Emily H. Thomas and Nora Galambos) ; (15) Linking Learning Outcomes to 

Student Course Evaluation (Stephen W. Thorpe) ; (16) The Effect of First-Year 

College Experiences on Student Persistence: A Case Study (Lillian Zhu) ; and 
(17) Outcomes Assessment and Re-Accreditation Data Concerns? Look to National 
Surveys for (Some) Help (Marianthi Zikopoulos) . (Each paper contains 
references. Contains 63 tables and 9 figures.) (SLD) 
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28 th Annual Conference 
PROCEEDINGS 

ERRATA 



Pages iii-iv Due to problems with the electronic transfer of the Proceedings document, the 
pagination was thrown off. Also, Karl Boughan's first name was incorrectly 
printed as Carl and Heather S. Roscoe and Michael E. Whitcomb were not 
included among the authors of the papers starting on pages 35 and 121, 
respectively. I regret the inconvenience these errors may have caused. The 
necessary corrections appear on the attached revised Table of Contents. 

Page 1 19 The authors of the paper titled "The Impact of Lottery Incentives on Student 

Survey Response Rate" are Stephen R. Porter and Michael E. Whitcomb. 
Michael's name was inadvertently not included. 



Dear NEAIR Friends and Colleagues, 



The 2001 annual conference of the Northeast Association for Institutional Research was held from 
November 17th to the 20th at the beautiful Hyatt Regency Hotel in Cambridge, Massachusetts. The 
inspiration and spirit of the conference is captured in the theme, "Institutional Research: Leadership 
through Excellence." Around 246 conference attendees shared in the celebration of our association and 
profession. With many innovative features, the conference explored how institutional researchers can 
become leaders in higher education by effecting policy changes through quality research studies. 

Highlights of the conference program include a stimulating opening keynote address, Increasing 
Institutional Researchers' Influence by Doctor Allan Cohen, Distinguished Professor of Global 
Leadership at Babson College. Jamie Merisotis, President of the Institute for Higher Education Policy, 
offered a thought-provoking plenary session, Turning Higher Education Research into Results: A Policy 
Perspective. The program also offered two excellent panel presentations - Beyond Traditional IR: 

Allies in Higher Education Research, convened by John Pryor, and Careers in IR: (and beyond?), 
moderated by Jennifer Brown. These events, and an impressive array of poster sessions, research 
papers, table topics, workshares, and workshops contributed to a very enriching program. The 
Proceedings represent some of the intellectual content and insights shared during the conference. 

The 28th annual conference marked the culmination of many months of creative planning and dedicated 
service by our Boston conference planning team. Our Local Arrangements Chair, Bea Frain, and 
Program Chair, Kelli Armstrong, deserve special recognition and admiration for their tireless team 
effort. Bea Frain's elegant taste, extraordinary organization, and attention to detail were evident in every 
aspect of local arrangements, especially in the delicious dinners and receptions, beautifully arranged 
conference settings and seamless conference scheduling. Thank you, Bea, for a superb job! 

As reflected in these proceedings and throughout the conference, Kelli Armstrong successfully created a 
high caliber program with rich offerings appealing to new members of NEAIR as well as to seasoned 
professionals. Kelli's qualities - her sensitivity to each person, flexibility with competing demands, and 
creative vision for the profession - contributed to a unique and memorable program. 

In addition to recognizing the conference chairs, I would also like to thank Ron Bentley who provided 
the wonderful jazz entertainment; Kathy Keenan who contributed her considerable artistic talents to the 
conference program; each member of the steering committee who dedicated hours to planning the 
conference; Marge Wiseman, Phyllis Fitzpatrick and our many volunteers who offered time and service 
during the conference; and each person who attended the conference. On behalf of all NEAIR members, 
I would like to express gratitude to Beth Simpson, our Membership Secretary, for her service during the 
year and for invaluable support with registration during the conference. 

Finally, I want to express my gratitude to Marianthi Zikopoulos, our Publications Chair, for the many 
hours she has spent in preparing this exceptional document. Current and future members of NEAIR will 
benefit from her careful work and professional expertise. In content and form, these Proceedings reflect 
the goal of our conference to influence policy through excellence in research. 



Anne Marie Delaney 
NEAIR President, 2000 - 2001 
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THE ROCKY ROAD TO GRADUATION: 

AN ACADEMIC CAREER FLOW MODEL 
FOR TRACKING STUDENT PROGRESS AT COMMUNITY COLLEGES 



Karl Boughan 

Director of Institutional Research 
Prince George's Community College 



INTRODUCTION 

During the last several decades, educational researchers have tended to answer the 
question “Why do some college students succeed and others fail?’’ by creating 
causal/predictive models of academic outcomes resting mainly on social psychological 
interpretations of collegiate behavior. Seminal examples are Tinto’s social/academic 
integration model (1987, 1993), Astin’s social input/academic output model (1977, 1993) 
and Bean’s economy of organizational incentives model (1982). Unfortunately, while this 
approach may be appropriate in the development of a general theory of student academic 
progress, it provides educational planners with very little practical information. 

When college administrators ask the crucial question, they are likely to mean 
“Where in the academic process are we losing students?’’ (e.g., during basic skills 
remediation, at the “gatekeeper” entry course point, while students are trying to satisfy 
their core curriculum requirements) and “Why are they dropping out at these points?” 
(e.g., poor academic performance, attendance fatigue, financial and family pressures). 
These practical queries are best answered by means of heuristic/descriptive models of 
student flow configured in terms of concrete academic processes. This paper presents 
such a model — one which maps enrollment flow in detail through (and out of) the 
academic process of a large, suburban community college, according to which student 
academic career phases coordinate with instructional program sequences. Its utility as a 
tool in enrollment management and instructional planning is illustrated in a model-guided 
six- year outcome analysis of fall entry cohort data. 

Student Flow Models in Institutional Research 

At the core of each student flow model is a map explicitly portraying a network of 
academic process streams through which students move over time and a coordinate set of 
flow measures which gauge movement volume at and between stream junctures. 
Promoted by the National Center for Higher Education Management Systems, which 
wedded projective techniques to this basically descriptive methodology, flow models 
enjoyed modest popularity among institutional researchers in the 1970s and 1980s (e.g., 
Johnson, 1974; Young, 1982). NCHEM-inspired models were created mainly as practical 
planning instruments for predicting such things as future course demand, degree major 
enrollment, and facilities usage. Beyond primitive term-to-term retention analysis, 



however, little attention was paid to the potential of flow models as tools of student 
outcomes research. This oversight has been somewhat amended by the current spate of 
published research blending flow modeling with event probability modeling (e.g., Ronco, 
1995; Kelly, 1996; Guerin, 1997; DesJardins and Moye, 2000) to explore the timing of 
types of enrollment outflows, and also by a number of recent flow-based studies centering 
on the outcome effects of particular academic process components (see St. John et al., 
1991; Andrade 1999; Kramer and LaMar, 2000; Schartman and Rhee, 2000). 

From a comprehensive planning perspective, the main problem with the former 
studies is that they focus too narrowly on outcomes timing issues, reducing student 
academic careers to a simple traversal of semesters. Academic process structure and 
operation, the factors most interesting to college planners, are either ignored in model 
construction or appear only as auxiliary control factors. The process component studies, 
on the other hand, tend to be more descriptive of process structure and operation and are 
therefore more useful to assessment and planning efforts, but their limited scope provides 
only small glimpses rather than the grand vista needed for full-scale strategic planning. 
The investigative lens for capturing the panorama properly must give a field of vision 
both wide and deep for obtaining an encompassing yet textured image of the whole 
academic process. 



Constructing a Student Academic Career Flow Model 

A student flow model capable of locating and measuring major enrollment 
management problems must be crafted to reflect the fundamental linked phase structure 
of the instructional process as it affects and is experienced by those undergoing it. In the 
case of most community colleges, these are easy to identify. 

Commonly, community colleges, as non-selective admissions institutions, have 
needed to craft comprehensive “developmental” pregrams to bring large proportions of 
their student bodies deficient in basic English language and mathematical skills up to 
acceptable levels of credit course-taking preparedness. Then, for completing remedial 
students and those entering as college-ready, a “general education” phase begins. This 
usually consists of two subphases: First comes a set of “gateway courses” in English 
composition and postsecondary mathematics (designed to hone the key communications 
and analytic skills students need for success in tackling subsequent college subjects). This 
is followed by general education proper — a regimen of entry-level courses in a 
prescribed variety of fields (representing the range of human knowledge and intellectual 
endeavor and providing the grounding for the in-depth study of particular disciplines). 
Given the educational logic distinguishing them, in this study we will treat gateway 
courses and the general education residuum as two separate, closely linked academic 
phases. Lastly, in cumulative phased learning terms, students round off their academic 
careers by developing a level of expertise in one academic discipline or occupational area. 
This is accomplished by working through a particular degree program of specialized 
courses — the “degree concentration” phase. 
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In addition to the above four instructionally defined phases (developmental, 
gateway, general education and degree concentration), one can posit a chronologically 
defined phase cutting across the others. It is no news that the great majority of U.S. 
postsecondary educational institutions experience disproportionately high early 
enrollment attrition rates. Prince George's Community College is no exception in this 
regard, typically losing around a quarter of its first-time college entry students between 
the initial fall and spring semesters. Many studies interpret this almost universal 
phenomenon as “college shock” — new student difficulties of adjustment to the 
unexpected rigors of college study and the alianness of college folkways. Since such a 
social psychological syndrome would constitute independent negative enrollment force 
acting independently of any academic process enrollment effects, we felt it prudent 
somehow to represent it in our student flow model as a separate dimension. In practice, 
this required splitting the developmental course-taking phase into preliminary first fall 
retention and remainder term subphases, and inserting a full non-developmental student 
first-fall retention phase before the gateway course-taking phase. 

The fact, however, that the instructional apparatus found at most community 
colleges, including PGCC, was meant to work as a progression of course types (from 
general skills to general knowledge to specialized knowledge instruction) does not, of 
course, guarantee that it actually functions this way. Indeed, the PGCC catalog presents 
the developmental, gateway, general education and degree concentration study 
components as simple graduation requirements to be completed in no stipulated order. 
Also, practical considerations of course load and class scheduling often necessitate cross- 
phase course enrollments. 

Even so, we find that most student course-taking does tend overall to follow the 
intended trajectory. The reasons are several: (1) Most students avoid random course- 
taking and seem naturally to order their academic careers in rough conformity with the 
education progression implicit in the college catalog, although they probably think more 
in terms of an “easy courses first” strategy than learning theory (the challenge posed by 
freshman English always seems to take them by surprise); (2) Academic counselors and 
faculty constantly prompt students to behave in reasonable compliance with the preferred 
study plan; (3) At PGCC, a fairly rigorous system of credit course pre-requisites promotes 
sequential phase course enrollment (for example, all gateway and most general education 
courses require prior completion of relevant remedial course work); (4) Students who 
habitually attempt advanced courses without proper grounding will naturally make little 
progress toward graduation and run a high risk of early dropout. 

All of the above act to shape a genuine system of sequenced instructional phases 
in which meaningful course work within an instruction phase is essentially restricted to 
students having completed all prior phases course work. The truth of this is strongly 
suggested by an attempted credit hour analysis we carried out on the Cohort 1995 data: 
For the first phase transition, students requiring but not finishing their remedial programs 
attempted only 2.2 credit hours worth of gateway courses, on average, compared with 8.3 
gateway credit hours for non-developmental and developmental completing students. For 
the second transition, the mean attempted number of general education credit hours 
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attempted by gateway phase non-completers came to only 5.1 against 13.9 attempted 
hours for completers. And in the third phase transistion case, general education non- 
completers averaged only 7.2 degree concentration credit attempts while general 
education completers accumulated an average of 36.1 attempted hours. Furthermore, only 
66 out of 2,239 1995 Cohort students ever complete an instructional phase out of turn. 

These findings lend empirical credence to our basic approach. Enrollment flow, 
by and large, does seem to follow a sequence of instructional phases, at least at PGCC, 
and therefore may be modeled accordingly. Unfortunately, the research also made clear 
that such phase-based flow was imperfect. Although the volume seemed too small to be 
of real consequence in the academic careers of most Cohort 1995 students, out-of-phase 
sequence course work did occur often enough to present us with a vexing technical 
problem. In a properly constituted flow model, students must work completely through 
one flow phase before proceeding to the next. Our solution to this problem was to employ 
the following model-building rule: Formally discount any out-of-phase course work and 
treat the first uncompleted instructional phase as solely responsible for all sub-graduation 
outcomes. This modeling rule forces only the elimination from the model of trivial 
course-taking, and reinforces the prime purpose of the flow model, which is to identify 
the root sources of enrollment outflow. The resulting academic career model of student is 
displayed in Figure 1. 

Model flow, as shown, progresses from left to right, with the central set of linked 
ovals representing the main progress-toward-degree stream; all other shapes shown 
represent non-degree track circumstances and outcomes, cumulative six year outcomes or 
associate degree attainment. Flow begins with a college readiness assessment shunt 
(rectangle) which divides all newly admitted students into two streams according to the 
results of developmental placement testing and supplementary methods of basic college 
skill evaluation. Students found requiring remediation here are entered into one or more 
pre-credit programs for reading, English and math skills development (diamond-indicated 
Phase 0). Internally, the developmental phase is divided into an initial term sub-phase 
(0a) for gauging the enrollment effects of “college shock” and a post-first fall Sub-phase 
Ob (not separately shown). In the meantime, those requiring no remediation proceed 
directly to the their first term of credit course work (Phase 1, the first oval), and surviving 
that, return the next spring to continue their gateway course work (Phase 2) in entry-level 
English composition and pre -calculus mathematics (10 credit hours). Joining them for 
Phase 2 study are the Phase 0 students who completed their developmental programs and 
are now vetted as credit-course-ready. Thereafter, phase-completing students go on to the 
general education Phase 3 (normally 16 credit hours in five subject areas — humanities, 
social science, physical/biological science, computer literacy, and cultural diversity), and 
if successful here to Phase 4 degree program course-taking (35-38 credit hours worth of 
degree department and elective courses) and possibly graduation with an associate degree 
(end-of-process parallelogram). 
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The downward flows branching off from the main stream depict the “negative” or 
suspended outcomes associated with each instructional phase. The falling perpendicular arrows 
represent phase non-completers exiting PGCC without any apparent academic attainment 
(dropouts) and the down-slanting arrows portray non-completing non-attainers still in attendance 
after six years (continuers). Upward branching flows off the main phase stream indicate two 
varieties of “positive” attrition — non-completing students leaving the college after receiving 
occupational certificates (a kind of early, minor graduation) and those transferring without a 
degree from PGCC. 

As already mentioned, the model was operationalized by means of a six year transcript 
analysis of the academic careers of the cohort of Fall 1995 new native entrants (N=2,239). The 
initial cohort database, derived from official student records, contained exhaustive term-by-term 
information on developmental testing, placement and program progress, credit course enrollment 
and performance, degree program and achievement, and attendance and withdrawal patterns. In 
addition, this material was supplemented by full-coverage transfer behavior data supplied by the 
National Student Clearinghouse, assuring an extremely high rate of accuracy in the separation of 
transfer from dropout outflows. 

Student developmental phase history was captured by means of standard developmental 
status transcript flags, thoroughly cross-checked with original placement test scores, 
developmental course performance, and the presence of credit course prerequisite waivers which 
might retroactively affect official remedial status. Gateway and general education course-taking 
were modeled in accordance with 1995 college catalog specifications. However, participation in 
the vastly varying degree concentration phase was not independently modeled. Here we simply 
applied the survivor principle: Any student completing all prior phases was taken automatically 
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to be a participant in the final phase. This worked out well in practice, since no prior phase 
survivor proved after six years to have accumulated less that 27 earned credit hours (26 earned 
credit hours are required for completion of phase 3). 



Model Assessment and Findings 

In the remainder of this paper, we will explore how well the PGCC academic career flow 
model performed when put into operation with actual student transcript data, and what its 
behavior had to teach our program planners and enrollment managers. A well-functioning student 
flow model designed as a general-purpose device for describing enrollment through-put and 
diagnosing enrollment problems should be able to fulfill multiple needs. At a minimum, it would 
be capable of delineating the basic unfolding pattern of student enrollment histories, of 
pinpointing and assessing the seriousness of enrollment problems by academic career phases, of 
suggesting reasons for the enrollment erosion discovered, and of helping college administrators 
strategize programmed responses. 



Figure 2.Cohort 1995 Six Year Cumulative Student Flow by Academic Career Phases 




Admission Placement Dev./TI Exit Dev./Retained College-Ready Ready/TI Exit Gateway Gen. Ed. /Other Major & 

Courses Electives 



Academic Career Phases 



Figure 2, a stacked area chart, shows the model at work in portraying overall Cohort 1995 
six-year outcomes flow. The area shadings identify the main retention-to-degree stream (white) 
and various categories of enrollment outflow. Area proportions (percent of total cohort) represent 
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cumulative main stream or outflow volumes at each successive academic career phase point. In 
the admissions pre-phase, 100 percent of Cohort 1995 students are shown as formally on degree 
track, changing quickly to 24 percent as the skills assessment pre-phase almost immediately 
shunts the bulk of new students (76 percent) into the remediation pre-credit pool. At the Phase 0a 
point (first term developmental course work), dropout outflow begin with a gush of 19 percent, 
surging to a full 46 percent in Phase Ob (remaining term developmental course work). The 
dispersal of the remediation pool becomes total with the formation of a developmental completer 
pool (19 percent) and the start-up of the continuing student sub-stream and transfer student 
outflow (both 3 percent). Then, at the credit course-ready collection point, the developmental 
completion pool empties back into the main stream, swelling it to a volume of 43 percent. From 
here forward, the pattern reduces to a simple matter of progressive main stream drainage up till 
graduation when the cohort diminishes to near zero (6 percent continuing students). Dropout 
outflow increases hardly at all at the non-developmental student first term point (Phase 1) , but 
spurts to 61 percent during gateway course-taking (Phase 2), thereafter climbing slowly to a 65 
percent finale (degree concentration Phase 4). More importantly, the latter interval between 
Phases 2 and 4 sees a healthy growth in transfer-only outflow, reaching 23 percent in the end. 



Table 1. Fall 1995 Entry Cohort Six Year Outcomes i 


(N=2,239) 




Whole 

Cohort 


Outcomes 

Subsample 


Positive Outcomes 


29.6 


100.0 


All Transfers ** 


25.7 


87.0 


All Transfers to 4 Year Schools** 


19.1 


64.4 


All Academic Awards** 


6.7 


22.8 


Transfer to 2 Year School 


6.7 


22.6 


Transfer to 4 Year School 


16.1 


54.4 


Assoc. Degree + 4 Yr Transfer 


2.9 


10.0 


Associate Degree Only 


3.3 


11.3 


Occupational Certificate Only 


0.4 


1.5 


Continuing/No Award or Transfer 


5.9 


100.0 


Dropouts (by Phase) 


64.6 


100.0 


All Post- I s ' Term College Ready** 


17.3 


26.8 


All Developmental Dropouts** 


44.8 


69.4 


All 1 sl Term Dropouts** 


21.5 


33.3 


P4-Degree Concentration Courses 


1.3 


1.9 


P3-General Education Courses 


2.8 


4.3 


P2-Gateway Courses 


13.3 


20.5 


PI -1 st Term Only Non-Developmental 


2.5 


3.8 


POb-Post -1 st Term Developmental 


25.8 


40.0 


P0a-1 st Term Only Developmental 


19.0 


29.5 


*60 or more cumulative earned credit hours after six years 

**Overlapping summary outcome categories; All Academic Awards includes certificates; All Post-1 st 
Term College Ready Dropouts includes Developmental Completers 



Table 1 restates the results of six years of Cohort 1995 study in a final outcomes format, 
utilizing the student flow pattern findings to expand the dropout category (exiting PGCC without 
either award or transfer) so that it reflects occurrence by study phase. This application of the flow 



data turns an otherwise ordinary outcomes report into a clear and precise means for locating 
enrollment erosion in academic process space and for assessing the relative seriousness of the 
problem represented by each outflow point. 

According to the table, 30 percent of the students attained a “positive result” (transfer or 
some sort of academic award) after six possible years of PGCC attendance, while the bulk 
transfer without a degree (23 percent). Most of these were proper transfers to four-year schools 
(16 percent), but a significant minority made a lateral move to another community college (7 
percent). Only 7 percent actually persisted all the way to graduation (almost all of these were 
associate degree earners). Rounding out non-dropout results, 6 percent turned out to students still 
attending (Spring 2001) and working on at least one outstanding graduation requirement. This 
left almost two-thirds (65 percent) in the dropout classification. By far, the weightiest dropout 
subset were students who got stuck at the developmental phase (45 percent of the whole cohort; 
69 percent of all dropouts). The next biggest contributor to the dropout pool proved to be the 
gateway phase ; P2 exiting non-achievers represented 13 percent of all students and 21 percent of 
all dropouts. Only 4 percent of cohort students left defeated from either of the two advanced 
career phases (6 percent of all dropouts). Finally, we may note that, of the two first-term attrition 
phases, the developmental subphase produced a far greater proportion of dropouts (19 percent of 
the cohort; 30 percent of all dropouts) than the non-developmental student phase (only 3 percent 
of all students and 4 percent of all dropouts). 
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A third way of looking at our flow data is to examine outflow types and rates phase by 
phase. Since each phase represents exposure to a different set of courses and subject matter, 
presenting different types and levels of academic challenge, individual phase outcomes analysis 
contextualizes student enrollment behavior as responses to process-driven variations in the 
learning experience. This in turn may lead to the formulation of more fitting intervention 
strategies and more effective instructional reforms. Figure 3 embodies our phase outcomes 
analysis of Cohort 1995 behavior in stacked bar chart form. Each bar reflects the enrollment flow 
consequences of participating in the course work required in a particular phase, and therefore is 
based solely on data for students actually reaching that phase in their academic careers. 

Phase sample rate for all phase-relevant flow categories are shown (the full range 
includes degree-less transfer, transfer with an associate degree, associate degree only attainment, 
occupational certificate only attainment, phase completion and advancement to the next phase, 
continued attendance without phase completion, and dropping out). In addition, dropout and 
transfer flows are shown divided into sub-flows. Two sorts of dropouts are distinguished — 
those dropping out with poor academic records and exiters whose level of academic performance 
seemed sufficient to assure eventual graduation. Classification was based on placement on an 
additive scale involving three performance variables (Cumulative GPA, proportion of courses 
passed, and proportion of semesters in official good academic standing). The transfer division 
was based on whether the move involved a four-year school (progressive transfer) or a shift to 
another community college (lateral transfer). 

Perhaps the most productive way to approach the enrollment flow patterns of Figure 3 is to 
look for the ways academic success probabilities are conditioned by each study phase students pass 
through, with an eye on practical policy implications. From that angle, the main findings can be 
stated as follows: 

• In dropout probability terms, the first semester does seem to be a particularly dangerous period 
for PGCC students. The negative exiting tendency of developmental Term-1 students (POa) fell 
within the middle of the all-phases range (in fact, it came to less than half the rate exhibited by 
POb developmental students beyond their first fall). The dropout proclivities of non-remedial 
Term-1 students (PI) registered the second lowest dropout rate of any phase. The prominence of 
POa students in Tables l’s negative outcome report (30 percent of all dropouts) was mainly a 
product of this phase’s high participation rate (76 percent). 

• Negative outflow ranged from a high of 48 percent in POb (latter term developmental study) 
to lows of 10 and 12 percent for PI (Term 1 non-developmental student) and P4 (degree 
concentration students). But despite these variations, the best summary of the data is that the “at- 
risk” problem, to some significant degree, extended across almost the entire academic careers of 
PGCC students. No phase or phase types monopolized it and it was wholly absent from none. 
The clear policy implication is that the college needs to broaden its future retention efforts far 
beyond traditional first term and developmental interventions. 
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• Type of dropout (“good student” versus “poor student”) is also important, since each type of at- 
risk student requires a different sort of assistance and support. According to Figure 3, early phase 
negative outflow tended to feature Cohort 1995 students with serious academic problems while 
more advanced phase dropout was dominated by students with passing course records. The 
proven substantiality of the phenomenon of the adequately performing student, who abandons his 
or her academic career just short of graduating or transferring, is particularly distressing. The 
college has many academic support programs for the academically challenged but offers 
relatively little structured assistance to academically talented student who may be struggling 
against ill health, financial problems, and the competing demands of employers and families, the 
probable root causes of enrollment attrition in this group. 

• Figure 3 also provides some useful data on positive phase outflow. As is the case at a great many 
community colleges, so it is at PGCC — successful students tend strongly to transfer before 
graduating. Figure 3 shows that transfer-only likelihood among Cohort 1995 students was a 
function of the last academic career phase reached. Little such transfer occurred in the first term 
but, its frequency increased rapidly and monotonically over the post-first-term phase sequence 
(from POb 1 1 percent to a P4 52 percent). In fact, transfer tendency in P3 and P4 was so robust 
that it became, ironically, the principle cause of failure to graduate. This finding puts a premium 
on efforts to promote the value of the associate degree, to reinvigorate the transfer degree 
advisement process, and to improve and increase the number of the college’s transfer articulation 
agreements. 

• Finally, as in the dropout case, type of transfer turns out to vary systematically by academic 
career phase. Although overall transfer outflow was slight in the early phases, what there was of 
it was of the lateral sort. This suggests a “bail-out” motive for shifting enrollment (using transfer 
to another community college as an alternative to dropping out of higher education altogether). 
At the other end of student academic careers, four-year transfers predominated. Even so, 
discemable minorities of transfer exiters at gateway and general education points in their PGCC 
careers shifted to two-year schools (4 and 3 percent of phase students, and 26 and 12 percent of 
phase transfers, respectively), which raises the possibility the market forces might be at work. 

Conclusions 

The purpose of this paper has been to describe a new enrollment flow model, based on the 
real-world features of a typical community college academic process as experienced by students, to 
test the validity of that model in terms of fit with longitudinal cohort data, and to demonstrate the 
practical diagnostic utility of model-based outcomes analysis for enrollment management and 
program planning. Unfortunately, space limitations have precluded more than a somewhat sketchy 
presentation of the model’s nature and operation, and many of our empirical findings and practical 
observations had to be set forth in underdeveloped form. We hope, however, that we have managed 
at least to establish the high potential of our approach for policy-relevant institutional research. 

Furthermore, we very much consider our academic career model of student enrollment flow a 
work-in-progress rather than a finished product, and recognize that much needs to be done to 
substantiate its method and to improve its explanatory and prescriptive power. We are currently 
pursuing those ends in a number of ways. First, we are working to refine model component 
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configuration and are struggling to bring genuine catalog-specified submodeling to the complex 
degree concentration phase. Secondly, we are moving toward conferring on the model the power to 
describe and assess the details of within-phase enrollment flow so that college planners can get a 
more pointed sense of what program reforms are needed to enhance retention, given each phase’s 
internal dynamics. Thirdly, we are exploring model-based enrollment analysis by student body 
subgroups (full-timers and part-timers; academic, social and financial support program participants; 
employment status and family status groups; age, gender and race/ethnic groups). Lastly, it seems to 
us that the academic career flow model, now purely descriptive, could fairly easily be transformed 
into a predictive instrument by applying event history analysis to phase flows, a possibility we are 
actively considering. However, whatever the technical progress we might manage to achieve along 
these lines, what matters most to us is to have succeeded in showing, by means of the present model, 
however flawed or unfinished, the likely benefits to outcomes research of taking the nuts and bolts 
of the academic process into systematic account . 
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PROMOTING A GLOBAL PERSPECTIVE 
THROUGH INSTITUTIONAL RESEARCH 



Anne Marie Delaney 
Director of Institutional Research 
Babson College 

Purpose. The purpose of this paper is to demonstrate how institutional research can 
promote an enlightened global perspective by enhancing understanding of international 
students' characteristics, values and goals. The paper is based on a completed research 
study designed to identify similarities and differences between international and domestic 
students attending a private college in the northeast section of the Unites States. The 
ultimate goal of this study was to guide the design of programs that would encourage 
integration by fostering appreciation of common bonds and respect for differences. The 
major research questions addressed were: 

• First, how do international and domestic freshmen differ with respect to academic 
quality; interests and activities; perception of abilities; reasons for going to 
college; and life goals? 

• Second, what are the policy and program planning implications of these 
differences for admissions, marketing, academic management and student affairs? 

Review of The Literature. A review of the literature highlights the significant 
presence, potential influence and challenges associated with increasing numbers of 
international students attending college in the United States. Foreign students in U.S. regions 
and states have increased dramatically from 48,486 in 1959-1960 to 514,723 in 1999-2000. 
Further, from 1998 to 1999, foreign student enrollment has increased 4.8 percent nationally. 
(Chronicle of Higher Education, November 17, 2000). 

International students offer a wealth of benefits to our country. As Keith Geiger, Deputy 
Assistant Secretary for academic programs in the U.S., observes, their presence ensures that 
there will be a cadre of people around the world who have a profound understanding of the 
United States which will potentially lead to enhanced relationships and increased cultural 
ties. (Desruisseaux, 1999). There is often, however, a discrepancy between the potential for 
cultural enrichment and the reality experienced by international students. "On many 
campuses, often to the dismay of faculty, students from culturally parochial backgrounds 
quickly link socially with similar others. Even in these globally cosmopolitan contexts, 
American and international students generally live in separate societies, hardly brushing by 
each other on sidewalks" (Seymour & Messinger, 1995, p. 4). 

Previous research has identified various personal, social, cultural and educational 
differences as factors that may account for the lack of interaction between international and 
domestic students on U.S. college campuses. Tompson and Tompson (1996) reported that 
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international students identified developing a social network, dealing with a different 
language, and coping with different norms and regulations as the most critical and difficult 
aspects of adjustment. Faculty cited little class participation, not asking for clarification, and 
sitting and studying only with international students as behaviors that undermine 
international students' academic performance. 

Three studies identified educational differences that may affect relationships between 
international and domestic students. Hamilton (1979) reported that international students 
perceived the university environment to be more competitive, the competition for grades to 
be more intense, and the professors to be more demanding. Boyer and Sedlacek (1986) 
discovered that international students have a different perspective on their education and the 
university environment. They take their education quite seriously, valuing it both for the 
intrinsic reward of academic pursuit and for career-related reasons. Recently, Ladd and Ruby 
(1999) found that international MBA students' adjustment involved shifting from the lecture 
method to a free learning environment in which they had to solve problems rather than 
memorizing facts. Students needed to be more independent in their approach to learning 
instead of relying on their professors. 

Cultural differences may also affect interaction between international and domestic 
students. Moline and Hendel (1992) found that, compared with students for whom English 
was their native language, non-native speaking students put more emphasis on their 
parents' wishes and the desire to gain a general education as reasons for going to college. 

They were more interested in being successful in their own business, helping others in 
difficulty, and promoting racial understanding. In another study (Davis et al., 1993), female 
international students attending an American university identified lack of attention to 
family, lack of appreciation for the arts, and relatively low interest in reading in the United 
States as significant cultural differences compared with the values of their country. 

Data Source. This study is based on analyses of trend data from the Cooperative 
Institutional Research Program (CIRP) survey - a national, longitudinal survey of first- 
time, full-time freshmen in the United States. The survey is sponsored by the American 
Council on Education (ACE) and the Graduate School of Education and Information 
Studies at the University of California, Los Angeles. The data set included cases for 1,000 
freshmen in the 1997, 1998 and 1999 freshman classes at the study institution. 

Methods of Analysis. Two-way analyses of variance were used to investigate 
significant differences by year, citizenship, and the interaction of year and citizenship on: 
hours spent in various activities during last year of high school, students' academic quality, 
self-ratings, reasons for going to college, and future goals. Discriminant analysis was 
employed to identify the most significant differences among international and domestic, 
male and female students. 
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Results 



Results revealed statistically significant differences by year and citizenship in several 
areas, including: typical high school activities, academic quality, student self-ratings, 
reasons for going to college, and life goals. 

Differences in Activities and Interests . Results revealed an interesting pattern of 
differences between international and domestic students in terms of time spent in activities 
during the last year in high school. As illustrated in Figure 1A, international students spent 
significantly more time studying and doing homework (F = 27.74, 
p < .001); reading for pleasure (F = 21.22, p < .001); praying or meditating 
(F = 6.97, p < .01); and talking with a teacher outside of class (F = 6.38, p < .01). 



Figure 1A. Significant Differences by Citizenship in Time Spent in 
Activities During Last Year in High School 



Score 

1 2 3 4 5 6 



Studying/Homework 
Reading for Pleasure 

Prayer/Meditation 

Talking with Teacher 
outside Class 





■ US □ International 



Note : Means are based on an 8 point scale from 1 ‘None’ to 8 ‘Over 20 hours per week’. 



In contrast, as shown in Figure IB, domestic students spent significantly more time 
working for pay (F = 68.73, p < .001); exercising or playing sports ( F = 41.48, p < .001); 
doing housework or caring for children (F = 19.03, p < .001); and socializing with friends 
(F = 10.41, p < .001). These differences regarding how students spent their time in high 
school are important for student programming as they suggest that students will prefer 
different types of activities during college. 
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Figure IB. Significant Differences by Citizenship in Time Spent in 
Activities During Last Year in High School 



Score 



1 



2 



3 



4 



5 



6 



Working for Pay 




Socializing with Friends 



Housework/Children 



Exercising or Sports 




6.06 



J 



■ US □ International 



Note : Means are based on an 8 point scale from 1 ‘None’ to 8 ‘Over 20 hours per week’. 

Variation in Academic Quality . Analyses of variance revealed statistically significant 
differences by year on two measures of academic quality. High school grade point average 
increased significantly from 1997 to 1999 (F= 3.95, p < .05). Admission test scores also 
increased significantly by about 20 points for all students during this three-year period. 

The SAT Verbal score increased from 569 to 580 (F = 8.82, p < .001) and the SAT Math 
score increased from 621 to 643 (F = 4.52, p <_.05). 

With regard to the primary focus of this study, significant differences were found by 
citizenship status on the SAT Verbal score (F = 68.44, p < .001). The domestic score of 
588 significantly surpassed the 531 score for international students. Analyses also revealed 
significant interaction effects on high school grade point average (F = 3.54, 
p < .05) and the SAT Verbal score (F = 4.44, p < .05). On high school grade point average, 
the domestic mean decreased slightly from 6.19 to 6.16 while the international mean 
increased substantially from 5.67 to 6.41. On the SAT Verbal score, the domestic mean 
increased somewhat from 579 to 594, while the international SAT verbal mean increased 
substantially more from 508 to 566. These results indicate that international students' 
academic quality has improved at a faster rate than that of domestic students. 



Students' Self Ratings of their own Abilities . Both international and domestic 
students' self-ratings increased significantly by year on the following four characteristics 
related to self-confidence and specific academic abilities: mathematical ability (F = 4.07, p 
< .05), intellectual self-confidence (F = 4.82, p < .01), social self-confidence (F = 3.95, p < 
.05), and writing ability (F =3.46, p < .05). 

Analysis by citizenship revealed that domestic students rated themselves significantly 
higher on several characteristics and abilities including: academic ability 
(F = 12.58, p < .001), competitiveness (F = 14.12, p < .001) and drive to achieve 
(F = 7.59, p < .01). In contrast, international students rated themselves higher on 
spirituality (F = 13.14, p < .001) and understanding of others (F = 6.49, p < .05). 



Figure 2 

Significant Differences by Citizenship in Students’ Self Ratings 

Score 

3 3.2 3.4 3.6 3.8 4 4.2 



_I L_ 



_J | 



Academic Ability 
Competitiveness 
Drive to Achieve 
Popularity 
Writing 
Leadership 
Spirituality 
Understanding of Others 



1 4.02 



J 3.83 



4.00 



_J 3.82 



P " 416 




3.55 



3.87 



i= 



J 3.73 



3.09 



3.43 



13.83 



3.99 



I US □International 



Note : Means are based on a 5 point scale from 1 ‘Lowest 10%’ to 5 ‘Top 10%’. 
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How students perceive themselves bears important implications for curriculum 
development and student program planning. These data identify perceived strengths and 
weaknesses and indicate areas that should be addressed in educational and student affairs 
programs. 

Variance in Reasons for Going to College . International and domestic students 
differed significantly on reasons for going to college. As illustrated in Figure 4, domestic 
students attributed more importance to getting a better job (F = 31.05, p < .001) and 
making more money (F = 21.00, p < .001). In contrast, international students placed more 
importance on gaining a general education (F = 7.76, p < .01) and their parents’ wishes (F 
= 5.12, p<. 05). 



Figure 3. Significant Differences by Citizenship in Reasons for 

Going to College 

Score 

1.0 1.5 2.0 2.5 3.0 



Get a Better Job 



Make More Money 



Get Away From 
Home 

Gain a General 
Education 

Parents' Wish 



■ US □ International 




Note : Means are based on a 3 point scale from 1 ‘Not Important’ to 3 ‘Very Important’. 



These differences regarding reasons for going to college reflect student values and suggest 
that students will have different expectations from college. This information is important 
both for admission recruiting and undergraduate program planning. One implication is that 
an undergraduate program that offers a general education, as well as excellent career 
preparation, will be attractive to international students. 
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Significant Variation in Life Goals . Analysis by citizenship revealed that international 
students differ significantly from domestic students on several future goals. As displayed 
in Figure 4, international students attribute more importance to being successful in their 
own business (F = 28.94, p < .001), developing a meaningful philosophy of life (F = 5.79, 
p < .05), helping others in difficulty (F = 23.27, p < .001), and promoting racial 
understanding (F = 17.86, p < .001). Some of these life goals are consistent with 
international students' emphasis on going to college to gain a general education and may 
indicate different expectations from the college experience as a preparation for life. 



Figure 4. Significant Differences by Citizenship in Goals 

Score 

1 1.5 2 2.5 3 3.5 



Be Successful in Own Business 
Meaningful Philosophy of Life 
Help Others in Difficulty 
Promote Racial Understanding 
Environmental Clean-Up 
Achieve in a Performing Art 
Create Artistic Work 





3.50 



4 



Note : Means are based on a 4 point scale from 1 ‘Not Important’ to 4 ‘Essential’. 



Results from Discriminant Analysis. To provide more information, the analysis was 
expanded to include gender as well as citizenship status. Discriminant analysis was used to 
predict membership in one of the following four groups: male U.S. citizens, female U.S. 
citizens, male international students, and female international students. Two statistically 
significant functions were identified. 



Table 1A displays the means for each of the four groups on function 1. Table IB 
identifies several variables correlated with function 1. As shown, function 1 contrasts male 
U.S. citizens with U.S. females and international males and females. A review of the 
group means on this function shows that U.S. males are most dissimilar to international 
females. It is also interesting to note that U.S. females are more similar to international 
students than to U.S. males. The statistical results on function 1 indicate that U.S. male 
students are typically high ability, confident, competitive students who share the desire to 
make more money as an important reason for attending college. In contrast, U.S. females 
and international students share more spiritual values. They are more interested in helping 
others in difficulty, in promoting racial understanding, and in achieving in a performing art. 
Further, parents' wishes and a desire to gain a general education are important reasons for 
U.S. female and international students' decision to attend college. 

Table 1A. Summary of Canonical Discriminant Function 



Function 1 


Group Centroids 


Male U.S. 


+.50 


Male International 


-.51 


Female U.S. 


-.42 


Female International 


-1.32 



Chi-Square = 394.66; df = 84; p < ,001 



Table IB. Structure Correlation Coefficients with Function 1 



Variables 


Correlation Coefficient 


SAT Verbal Score 


.52 


Competitiveness 


.47 


Leadership Ability 


.27 


Intellectual Self-Confidence 


.29 


Math Ability 


.27 


Academic Ability 


.27 


SAT Math Score 


.25 


Desire to Make More Money 


.21 


Emotional Health 


.21 


Spirituality 


-.22 


Reason for College- Parents’ Wish 


-.25 


Reason - Gain a General Education 


-.27 


Help Others in Difficulty 


-.28 


Promote Racial Understanding 


-.29 


Achieve in a Performing Art 


-.30 



Function 2 contrasts international students with U.S. citizens, particularly U.S. male 
students. Table 2A displays the means for each of the four groups on function 2. Table 
2B identifies several variables correlated with this function. According to these data, 
international students perceive themselves as more spiritual and creative. They are more 
confident intellectually and are more interested in the goal of being successful in their own 
business. Also, U.S. male students, compared particularly with international male students, 
have higher SAT verbal scores, higher average high school grades, and are more focused 
on attending college to get a better job. 

Table 2A. Summary of Canonical Discriminant Function 



Function 2 


Group Centroids 


Male International 


+.97 


Female International 


+.49 


Female U.S. 


.009 


Male U.S 


-.65 



Chi-Square - 170.14; df - 54; p < ,001 



Table 2B. Structure Correlation Coefficients with Function 2 


Variables 


Correlation Coefficient 


Be successful in own business 


.48 


SAT math scores 


.38 


Intellectual self-confidence 


.23 


Creativity 


.23 


Self-understanding 


.21 


Achieve in a performing art 


.18 


Spirituality 


.11 


Reason for College - Get a better job 


-.23 


SAT verbal scores 


-.26 


Average high school grades 


-.35 


As shown in Table 3, the discriminant analysis accurately predicted 64 percent of all 
cases. By far, the highest prediction rate is for U.S. male students. In contrast, the 
prediction rate is quite low for international students, particularly male students. 


Table 3. Classification Results: 


Predicting Group Membership 


Percent Correctly Classified 


Male U.S. 


86.8% 


Female U.S. 


43.9 


Male International 


16.9 


Female International 


32.3 


All 


64.1% 
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Results from this analysis suggest the need for more information to better understand 
international and domestic female students. Further research should seek to identify their 
unique qualities, values, aspirations, and goals. Insight gained from such research will 
potentially enable the College to better serve these students and to promote greater 
interaction among international and domestic male and female students on campus. 

Discussion 

Results from this research advance our understanding of significant differences 
between international and domestic students. The following discussion relates findings 
from this study to results from previous research. Knowledge from this body of research 
provides a basis for planning programs to promote enhanced relationships and to realize 
the potential benefits of international students' presence at colleges in the United States. 

In their study on international students' adjustment, Tompson and Tompson (1996) 
found that developing a social network and coping with different norms were among the 
most difficult aspects of adjustment. Results from this study suggest that differences in 
interests, goals and values may account for some of the adjustment difficulties. Compared 
with domestic students, international students spent more time reading, studying, praying 
or meditating and conversing with their teachers; whereas domestic students spent 
significantly more time working for pay, exercising or playing sports, and socializing with 
friends. These differences are important as they suggest that students will prefer different 
types of activities during college. 

Similar to findings reported by Moline and Hendel (1992) and Davis, et al. (1993), 
international students in this study also expressed greater interest in various social, 
philosophical and artistic goals, including developing a meaningful philosophy of life, 
helping others in difficulty, and achieving in a performing art. Recognizing and 
understanding these differences in values is essential to developing programs that will be 
of interest to international students and to fostering meaningful relationships between 
international and domestic students. 

How students perceive themselves also affects relationships. Analysis by citizenship 
revealed that domestic students rated themselves significantly higher on several 
characteristics and abilities, including: competitiveness, drive to achieve, and academic 
ability. In contrast, international students rated themselves significantly higher on 
spirituality and understanding of others. 

Similar to results from previous studies (Boyer & Sedlacek, 1986; Moline & Hendel, 
1992), findings from this research also show that international and domestic students have 
different reasons for going to college. Domestic students attributed more importance to 
getting a better job and making more money. In comparison, international students placed 
more importance on gaining a general education and their parents’ wishes. 
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Results from this study were translated into strategic policy recommendations for the 
President's Cabinet, the Undergraduate Admission, Academic Program Management, and 
Student Affairs Departments at the study institution. The overall goal of these 
recommendations was to enlighten future policy and guide the design of programs that 
would encourage integration by fostering appreciation of common bonds and respect for 
differences. Illustrative recommendations follow. 

Recommendations 

1. In recruiting international students, emphasize the College's ranking in 
entrepreneurship. 

These data indicate that international students are significantly more interested than domestic 
students in being successful in their own business. 

2. Design undergraduate admission publications and communications to reflect 
international students' intellectual, cultural and social values. 

Results from this study reveal that international students spent significantly more time 
studying, reading for pleasure, and praying or meditating during high school, and they report 
significantly higher interest in certain personal and social goals: developing a meaningful 
philosophy of life, helping others in difficulty, and promoting racial understanding. 

3. Emphasize opportunities for students to communicate with the faculty. 

Compared with domestic students, international students reported they spent 
significantly more time talking with a teacher outside of class in high school. Such experience 
would potentially lead them to expect similar opportunities during college. 

4. Portray the College experience as an opportunity to gain a general education as 
well as to prepare for a career. 

In discussing their reasons for going to college, international students placed 
significantly more importance on gaining a general education while domestic students 
attributed more importance to getting a better job and making more money. 

5. Design programs to identify and serve students in need of language support. 

Overall, international students' SAT verbal scores were significantly lower than those 
of domestic students. While the international student mean score increased from 1997 to 1999, 
it still remained lower than that of domestic students. 
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6. Offer seminars and discussion groups on culture and values for international and 
domestic students to learn about each other's cultural values. 

Results from this study reveal that international and domestic students differ 
significantly in their reasons for going to college and in their interest in pursuing many 
personal and social goals. Some of these differences may reflect differences in cultural values. 

Offering students an opportunity to learn about different cultures may foster greater 
understanding and ultimately promote increased interaction between international and 
domestic students. 

7. Develop social programs, highlighting different international cultures, to ensure 
that international students feel welcome and domestic students learn to 
appreciate the values and customs of international students' countries. 

Findings from national studies reveal that American and international students often 
live in separate societies while attending the same college. The opportunity for cultural 
enrichment associated with international students' presence is often missed. 

8. Evaluate the undergraduate program to ensure that it provides international 
students with opportunities to achieve growth in a broad range of life goals. 

Comparative analyses identified statistically significant differences between 
international and domestic students on several goals, including: developing a 
meaningful philosophy of life, helping others in difficulty, and promoting racial 
understanding. These life goals are consistent with international students' emphasis on 
going to college to gain a general education and may indicate different expectations 
from the college experience. 

9. Undergraduate program and course planning need to take into account the 
increase in student quality from 1997 to 1999. Special attention should be given to 
ensure that academic courses and programs keep pace with student quality. 

From 1997 to 1999, admission test scores increased significantly among entering 
freshman classes. Scores increased by about 20 points from 569 to 589 on the SAT 
Verbal score and from 621 to 643 on the SAT Math score. 

10. The College should continue to monitor international student trends and examine 
the extent to which international and domestic students interact and enrich each 
other's lives. 

While the presence of international students offers the potential for cultural enrichment, 
experience suggests that the College may need to exercise initiative to realize this 
potential. 
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Abstract 

In 1999-2000, Penn State implemented a first-year seminar requirement for all incoming 
baccalaureate freshmen, in all majors, at all Penn State campuses. This paper describes the role 
that institutional research played through three stages of the process: 

• early committee deliberations; 

• implementation of the new program; 

• support and development of the program. 

Student and faculty focus groups and surveys have provided mostly positive feedback 
about the seminars, along with some insights into opportunities for refinement and improvement. 
Analyses of non-obtrusive data such as transcript files and registrar's databases have been an 
especially efficient and powerful way to answer questions about how the new requirement has 
been enacted - To what extent are students fulfilling the requirement? Through which courses? 
For how many credits? In which colleges? - and so on. 

At Penn State, we have emphasized relatively pragmatic analyses (such as how the 
curriculum is being enacted) more than classic assessment approaches ( such as pre- and post- 
tests), for two reasons. First, the challenges of conducting valid and reliable assessments of 
cognitive and affective gains , or educational outcomes, are substantial, especially for campus IR 
staffs juggling multiple responsibilities with limited resources of time, staffing, money, and 
expertise. Second, a persuasive research literature has demonstrated that first-year seminars do 
constitute good practice. Therefore, our own efforts have mostly been directed toward 
developing, supporting, and strengthening a program that faculty and administrators believe is a 
good idea. We suspect that these probably are legitimate considerations for many of our 
colleagues in institutional research offices at other campuses, as well. 

Introduction 

Beginning in 1999-2000, Penn State University initiated one of the nation’s most 
ambitious first-year seminar efforts when it enacted a university-wide first-year seminar 
requirement for all new incoming baccalaureate students. Institutional research was 
involved in all phases of the design, implementation, and assessment of the first-year 
seminar program. This paper draws from that experience to illustrate how institutional 
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researchers can contribute to this type of curricular change. 

Origins of the First- Year Seminar Program at Penn State 

The first-year seminar program at Penn State was part of a larger change in 
general education at the university. In December 1997, Penn State’s Faculty Senate 
adopted the recommendations of a special committee, which had been working for over a 
year to develop a new general education curriculum (Penn State, Fall 1997). 

Some of the changes in the general education package were more substantial and 
challenging than others, but in total, the adjustments were significant. The committee 
presented ten recommendations (all of which the Faculty Senate ultimately accepted) with 
the goal of enhancing curricular flexibility, emphasizing high quality, fostering 
opportunities for experimentation and building assessment into the curricular process. 
Among the committee’s recommendations were the inclusion of active learning elements 
in all general education courses and the identification of key competencies. The 
committee also recommended the restructuring of requirements for health and physical 
education and for foreign languages. (More detailed information is available at Penn 
State’s general education website, http://www.psu.edu/oue/gened/.) The hallmark of the 
new general education curriculum was the establishment of a first-year seminar 
requirement for all incoming baccalaureate students. 

Members of the general education committee certainly drew on their own 
experiences and ideas in developing their report, but more objective information from 
qualitative and quantitative analyses was also important at all stages of the process: in 
initial explorations into alternatives and possibilities, through the implementation of 
curricular changes, to the assessment of the new program (Dooris and Blood, 2001). 

The Role of Institutional Research 

Institutional research supported the committee with data from many sources 
throughout the three stages of early deliberation, program implementation, and program 
support and continuation. 

Early Deliberations about a First-Year Seminar Program 

Early in the discussion process, institutional research helped confirm the 
desirability of some type of program along the lines of a first-year seminar. For example, 
committee members learned from transcript analyses that 55 percent of general education 
student credit hours were earned in classes of 100 or more students and that freshmen and 
sophomores were disproportionately likely to be enrolled in large classes. Attendance 
studies revealed that student absenteeism was strongly and positively correlated to class 
size; alumni and student surveys highlighted a need for students to better see the linkages 
between general education with studies in the major, especially early in the college 
experience. Several colleges within the Penn State system were already using elective 
first-year seminars and data were available from the assessment of these existing 
programs. The literature on undergraduate education and on first-year seminars as well as 



benchmark information on practices at other colleges (not all of which were in Penn 
State’s normal research university peer group) was also helpful. 

First- Year Seminar Implementation 

Institutional research also played a very useful role during the implementation 
stage. The actual creation of a first-year seminar program is a significant undertaking for 
any institution; it is especially ambitious for an institution as large and organizationally 
complex as Penn State. Each year Penn State enrolls about 12,000 new first-year students 
on its 24 campuses. It has 17 undergraduate degree-granting colleges that offer majors in 
232 undergraduate programs. Every college or university is, of course, unique - but 
institutional research can help faculty, staff, and administrators at any institution decide 
whether and how a major curricular change can, as a practical matter, be designed and 
implemented in the context of practical and specific organizational considerations. 

In Penn State’s situation, the general education committee was interested in 
proposing a university-wide requirement, but had doubts about whether the university 
would be able to overcome some very significant obstacles. Institutional research helped 
address these doubts by developing information on factors such as existing patterns of 
faculty instructional assignment; curricular requirements (credit loads, course sequencing) 
of different majors; estimated numbers and costs of additional small sections; and 
physical facility constraints — that is, the number, size, and availability of classrooms. 

Taking such factors into consideration, the committee decided that an institution-wide 
program could work if it combined a university-wide philosophy with considerable college, 
campus, and departmental flexibility. 

University-Wide Approach 

The most important university-wide aspect is the fact that there is a Penn State first-year 
seminar requirement that applies to all entering baccalaureate freshmen, in every academic 
program of the university and at every campus. We believe Penn State is the only large 
research university in the nation with such an institution-wide provision. Also, the seminars 
are all credit-bearing, all taught by regular full-time faculty with at least three years teaching 
experience at Penn State, and all limited to twenty students per section. 

College and Department Flexibility 

While the program has strong university-wide common threads, the first-year seminar is 
nonetheless as varied and individual as Penn State’s many colleges, campuses, and 
departments. 

For example, the first-year seminar courses range from one to three credits. Most 
students take a seminar in the college in which they are enrolled, but they may select from a 
wide array of courses, and satisfaction of the requirement is portable. Colleges, departments 
and campuses design and offer their own courses and have considerable autonomy as to the 
content and structure of the offerings. In recognition of the difficulty in implementing a 
university-wide program to serve 12,000 students, the Faculty Senate and the University 
administration specified a two-year transition period between acceptance of the 
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recommendations in 1997, and enactment of the first-year seminar requirement for freshmen 
entering in the Summer and Fall of 1999. 

Assessment of the Program 

Institutional researchers have been and remain involved in assessment of the first- 
year program, and in reporting results annually to the Faculty Senate and to the deans’ 
council. 

Consistent with the idea that assessment should be designed into a new program, 
one of the ten recommendations of the general education committee in fact called for 
systematic assessment of general education. As a result, the Faculty Senate and the 
university administration appointed a nine-member General Education Assessment 
Interest Group. That group identified the assessment of first-year seminars as its first 
task; it helped to guide much of the institutional research described in this paper. 



The Need for Footprint Assessment 

First-year seminar programs are frequently asked to prove their value; this theme 
threads throughout the publications of South Carolina’s highly regarded National 
Resource Center for the First-Year Experience. 

There is often a temptation to focus such assessments on the most conceptually 
interesting research questions about gains, cognitive and affective outcomes, and the like, 
but every college or university should probably first ask, “To what extent is this 
institution actually enacting its supposed curricular requirements? What is the footprint of 
programs, in terms of offerings and course-taking patterns?” Information about course 
offerings and student enrollment patterns are prerequisite to an evaluation of the strengths 
and weaknesses of a program. This is particularly true in the case of Penn State, given 
the myriad of first-year seminar implementation models among the individual colleges 
and campuses. Fortunately for institutional researchers, data about these patterns are 
available unobtrusively from transcript data, registrar’s files, and the like. Increasingly, at 
many colleges and universities, such data can be accessed fairly easily from a central data 
warehouse. 

The Penn State Data Warehouse 

In 1994, Penn State began development of a university-wide data warehouse. The 
aim of the data warehouse was to simplify ad hoc access to the most widely used 
administrative data. 

The data warehouse has since grown to approximately 100 tables in more than a 
dozen databases housing just under 100 million records. Data are transferred from the 
legacy systems on a regular schedule. These data are non-modifiable and represent a 
snapshot of time-fixed data. The Penn State data warehouse provides a convenient and 
consistent source of institutional data and allows for ad hoc inquiries as well as extensive 
analysis. 
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Using the data warehouse - specifically the data available in the university’s 
official enrollment, term course master, and transcript files - we were able to describe 
how the first- year seminar requirement had been enacted. For example, by the end of the 
1999-2000 academic year, 93 percent of the first-time enrolled freshmen who returned as 
sophomores had completed the first-year seminar requirement. There were 764 sections 
of 234 different courses offered to satisfy the requirement. The sizes of the sections 
ranged from 14 to 25, with most sections within the desired maximum size of 20 
students. The large majority (81 percent) of the courses were offered for one credit, eight 
percent of the courses were offered for two credits, and 11 percent of the courses were 
offered for three credits. 

Concerns over the university’s ability to implement such a significant curriculum 
change were eased, since these data also showed that more than enough first-year seminar 
places were available to serve the needs of the student population. The data also have 
helped to guide the development of procedures to handle situations in which a student 
does not satisfy the requirement in his or her freshman year. 



Student and Faculty Views on the First-Year Seminar Program . 

Penn State relied largely on faculty and student focus groups and student surveys in 
gauging the impact of the first-year seminar program and in identifying areas where 
improvement is needed. 

Faculty Focus Group Results. Faculty focus groups representing ten colleges and four 
campus locations provided feedback that was mostly positive. A strong message was that it 
is desirable to allow faculty to be creative with the content and structure of their individual 
first-year seminars while still ensuring that common objectives are addressed. Almost all the 
faculty members expressed a desire to teach a first-year seminar again. 

Student Focus Group and Survey Results. Focus groups were conducted with students 
from six colleges and four locations, and approximately 500 freshmen who had 
completed or were currently enrolled in first-year seminars were surveyed via written 
instruments. As with the faculty feedback, student reactions were largely positive. 
Students especially liked the small class sizes, which created opportunities for interaction 
both with faculty and with other students, many of whom were also enrolled in their other 
courses. Students identified time management skills, academic content, career knowledge 
of major and field and enhancement of library and Internet or computer skills as the most 
important things they learned in the first-year seminar. 

Surveys also found that almost half of the respondents felt that their first-year 
seminar resulted in their becoming engaged in the climate of learning at Penn State, being 
oriented to high expectations and demanding workload of academic life, and seeing a 
connection between the first-year seminar and their potential major. 35 percent of survey 
respondents felt that interacting with their first-year seminar instructor “added to the 
quality of their first year experiences” (Penn State, 2001). 
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While there were no strong negative comments, student feedback did indicate 
some areas for improvement in the program. Students who had completed a compressed 
seminar reported that too much material was covered in too short a time. Students who 
had not taken the seminar in their first semester felt that they should have take it then. 
Students felt that they should have been informed earlier about the array of first-year 
seminar choices available to them across the university. 



Conceptual, Methodological, and Practical Considerations 

Higher education now has about two decades of substantive, large-scale 
experience with first-year seminars, and with the assessment of those programs. Because 
this paper focuses on the assessment aspects of Penn State’s program, it is worth at least 
briefly discussing what the university did - and did not do - in the context of that 
broader, national experience. 

Penn State’s basic approach was essentially pragmatic. The faculty and 
administration strongly believed in the value of first-year seminars. They also felt 
assessment was important - but primarily to help create, implement, and continually 
improve a program that is fundamentally a good idea (versus, for example, to contribute 
to the scholarly literature on the topic, or to explore a subject of mostly academic 
interest). In this respect, Penn State was not very different from most colleges and 
universities. John Gardner, for example, has observed that the freshmen seminar 
movement has mostly been directed toward developing programs that work, and only 
secondarily to developing elaborate evaluations of those programs (Gardner, 2001). 

We believe that this is a very sensible bias for individual institutions, because the 
broad picture that has emerged from two decades of work is quite clear. 

First-Year Seminars as Good Practice 

First-year seminars constitute good practice. A compendium of studies from 50 colleges 
and universities provides evidence from institutions of all sizes, missions, and selectivity: 
“Retention rates improve, grades improve, students’ internal locus of control increases, 
participation in extracurricular activities and the use of campus services both increase, 
and students begin to clarify their short- and long-term goals. Most importantly, 
graduation rates increase” (Barefoot, 1998, p. xi). 

Reviews of numerous studies have shown that the first six weeks of the freshman 
year are the critical determinant of ultimate graduation (Erickson and Strommer, 1991). 
The evidence also shows that first- year seminars are an effective method for initiating 
students to higher education, helping them to make a successful transition from high 
school (Leamnson, 1999). Upcraft and Gardner’s (1989, pp. 4-1 1) review found that 
freshmen seminars enrich opportunities for student involvement - vital to freshman 
success - and that there is “conclusive evidence. . .that the freshman seminar is a very 
powerful way of enhancing freshman success.” 
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The Measurement Challenges 

Looking at the issue of evaluating excellence in undergraduate education, Ernest 
Pascarella (2001) recently reviewed some of the significant methodological challenges to 
validly and reliably measuring quality, excellence, or success. How would an institution 
decide what outcomes to measure? What particular set of competencies, activities, and 
accomplishments can be attributed to the undergraduate experience? How does a study 
control for out-of-class or out-of-college experiences? How does a study control for 
differences in student ability? And so on. 

Our strong suspicion is that few campus institutional research practitioners - 
while juggling multiple responsibilities with limited resources - have the time, money, 
staff, and psychometric expertise to overcome these research design challenges on a 
realistic schedule. 



Emphasize Good Practice. Bluntly put, at some level choices must be made between the 
sort of pragmatic implementation-oriented assessment that Penn State has emphasized, 
and more classic assessment approaches: pre-test/post-test, quasi-experimental designs, 
outcome measurements, and so on. 

Interestingly - because he was not writing about the assessment of first-year 
seminars or about institutional research - Pascarella did suggest an approach which is 
very consistent with most of Penn State’s institutional research on first-year seminars. In 
brief, Pascarella suggested a focus upon the practices and processes that are known to be 
linked to important cognitive and noncognitive outcomes. He wrote, “the assumption 
here, and it is not an unreasonable one, is that an excellent undergraduate education is 
most likely to occur at those colleges and universities that maximize good practices” 
(Pascarella, 2001, p. 22). 

John Gardner has noted that first-year seminars are “the most studied and assessed 
course genre in American higher education history” (1998, p. xiii). It is important for 
institutional researchers to take this observation seriously, and to help their respective 
campuses take advantage of the collective wisdom that higher education has accumulated. 

In short, first-year seminars are a good idea. Peter Ewell suggested (2001) that 
researchers continually ask themselves two concrete, core questions: “What happened?” 
and “What mattered?” We believe that is useful, wise, and legitimate advice for 
practitioners involved in institutional research on first-year seminars. 
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Introduction 

The use of the web in conducting survey research in institutional research is on 
the rise. The benefits to be gained are numerous: cost savings due to the elimination of 
copying, mailing, and data entry; “instant data,” which allows for quicker turn-around 
time; and easier access to a wider population, such as international respondents. While 
the verdict is still out on methodological issues such as response rates and biases, web- 
based surveys are the trend of the future. 

However, there is uncertainty among IR professionals on how to proceed in 
implementing web-based surveys. The questions are myriad: Do we build our own 
software or purchase an existing product? Do we need our own server, use a server on 
campus, or contract to use a private server? Do we need technical staff members in the 
IR office or can we do it ourselves? The list goes on. One thing is certain. Unless your 
IR office already has an IT or systems expert, you need to take the time to learn about the 
different options and what will work best based on the level of technical support, the 
complexity of the technical infrastructure at your campus, and your own level of technical 
expertise. 



Objective 

The primary objective of this paper is to describe the process of one IR office’s 
attempt to select web survey software, coordinate the use of on-campus servers and 
technical staff, and evaluate the effectiveness of the entire process. This case study will 
inform and educate other institutional research professionals considering a move from 
paper surveys to web-based surveys. 



Literature 

The body of literature on the methodological implications of web-based surveys is 
growing (Matz, 1999; Cartwright, Thompson, Poole and Kester, 1999; Underwood, Kim 
and Matier, 2000; Porter and Umbach, 2000). These and other studies discuss at great 
length the methodological costs and benefits of moving from paper to the web, and 
greatly inform the IR professional’s decision to make that move. The benefits, of course, 
are clear. It is much cheaper to conduct a web-based survey since postage, printing, and 
other distribution costs are not incurred. It is also much quicker to conduct a web-based 
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survey, since data is entered directly by the respondent to an electronic data file with out 
the need of second-party data entry. The costs of web surveying, however, may have an 
impact on survey results. Several institutional research projects in higher education 
reported lower response rates for web-based surveys as compared to paper surveys 
(Cartwright, Thompson, Poole and Kester, 1999; Underwood, Kim and Matier, 2000). 
Underwood et al also reported that underrepresented minority students responded on the 
web at a lower rate than White and Asian students. However, some non-higher education 
studies found no significant differences between paper and web survey responses (Smith 
1997). 

Currently, there are no studies that specifically address the decision-making 
process required for choosing the tools for web-based surveys and for dealing with 
campus technical-organizational issues. While some studies touch on logistical issues, 
such as Porter and Umbach (2000) and Cartwright, Thompson, Poole and Kester’s (1999) 
description of assigning students unique PIN numbers, there are no comprehensive 
descriptions of the decision-making process in regards to applying available technology to 
an IR office’s surveying needs. 




Surveying Background and History 

Description of Structures and Processes 

This case study describes the decision-making processes involved in 
implementing a web-based survey system by the Office of Institutional Research (OIR) at 
Tufts University. Tufts is a Doctoral/Research University-Extensive located just outside 
of Boston in Medford, Massachusetts. The university comprises eight schools: Arts & 
Science, Engineering; Fletcher School of Law and Diplomacy; Nutrition Science and 
Policy; Medical; Sackler School of Biomedical Sciences; Dental; and Veterinary. The 
Office of Institutional Research is a centrally-located unit, serving all of the schools of the 
university as well as the central administration. 

The OIR at Tufts is known across campus - to both administrative and academic 
units — as a good resource for conducting survey research. Assistance to units ranges 
from consultation on survey item development to management of entire projects. In 
fiscal year 2000, the time period immediately before web surveys were used by the OIR 
on a regular basis, the office administered 24 different surveys. These surveys were both 
annual projects, such as the senior survey and the accepted applicant surveys, and ad hoc 
or intermittent projects, such as the undergraduate alumni survey. They also ranged in 
size from a single page with an N of less than 100 to multi-page instruments with N’s 
over one thousand. The OIR at Tufts has a history of creating their own “homegrown” 
survey instruments rather than relying on standard national surveys. For example, the 
OIR, in collaboration with admissions staff, developed and administered a New Student 
Survey and A Non-Enrolling Survey instead of using the standard Admitted Student 
Questionnaire survey, which many other college campuses use. The theory behind the 
development of these homegrown instruments is that it gives the OIR’s client - the Office 
of Admissions - the ability to ask university and issue-specific questions to a greater 
degree and allows the OIR to have more control over data formatting. 

41 
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The processes in the OIR were somewhat standard for administering paper 
surveys with large populations, and return procedures varied based on the project. For 
example, business reply envelopes were provided for surveys that went off campus to 
alumni. Surveys on campus, to student or faculty populations, were either returned 
through campus mail or collected directly from the respondent. Once the completed 
surveys were returned, the data was entered manually, usually by student research 
assistants. The office experimented for a period of time with two different scanning 
procedures, but found that they were not efficient and the time and labor cost differentials 
compared to manual data entry were not great. 

The result of the OER’s success in cultivating a sense of trust and understanding in 
the importance of data collection to inform and improve programs among its campus 
clients was a dramatic increase the number and scope of surveys administered. This has 
been especially true since the mid-to-late 1990’s, when the evaluation and assessment 
movement took hold in higher education Along with this increase, however, came the 
need for resources to complete these research studies. The cost of most of these surveys 
ran anywhere from a few hundred dollars to several thousand dollars. The OER had to 
employ more and more student research assistants in order to assist with the 
administration of the surveys as well as the resulting data entry. 

Discovering the potential of a web-based system 

Several factors came into play as the OER determined that the development of a 
system to administer surveys on the web was a priority. First, as mentioned above, the 
office had used two different scanning mechanisms with the hope that data could be 
entered faster and reduce the need for manual data entry. The first scanning system 
(“system” meaning the use of specific software with a stand-alone scanner) was 
implemented in 1996, with the goal to scan many of the larger surveys while only making 
minor adjustments to the formats of the actual instruments. The procedure of 
reformatting the instruments turned out to be quite labor intensive for the research 
analysts, and the actual scanning turned out to be problematic, with much time spent on 
cleaning up the resulting data files. In 1998 a new scanning software was purchased 
along with a higher-grade scanner in order to correct the problems encountered with the 
previous scanning system. But, while the software was a bit more robust and the scanner 
faster, there was still an inordinate amount of time being spent on the back-end cleaning 
up data files. While the OER’s Executive Director and other staff members realized that a 
more cost effective way of administering surveys was important, it was determined that 
the old method of manual data entry was more efficient, and by the summer of 1999 there 
was virtually no scanning of surveys in the OER. 

Also, the staff of the OER was becoming aware of the viability of conducting web- 
based surveys through sources within the field of institutional research and through new 
technology being utilized externally, especially in the business sector. By 1999 the use of 
web-based surveys in institutional research was being discussed widely and actually being 
used by a few institutions. There seemed to be little consensus, however, on how 
effective it was (or would be) in addressing the data collection needs of an institutional 
research office. But, since the OER at Tufts had already been thinking strategically about 
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changing processes - as culminating in the scanning efforts - the idea of doing surveys 
via the web was taken seriously by the executive director and staff. 

The technology was improving quickly, and by this time commerce and data 
collection through the web was standard operating procedure by businesses. The 
advancement of Java and other graphical interfaces made it easier to format and collect 
data, and the improvements in security gave people a greater sense of confidentiality. 
But, most importantly, it made interaction user-friendly for the respondent. 



Selection of web-based software 

The vision the office had in selecting the software and hardware for web-based 
surveying focused on maximizing survey effectiveness and making the entire survey 
administration process efficient. To do this, the ultimate goals were to: 1) implement a 
system that was seamless, 2) to have control over and ensure the security of the process 
and resulting data, and 3) to have a methodologically sound process. 

Seamless. Generally, all quantitative data analysis in the OIR is conducted using one 
specific statistical analysis package. One concern that was persistent among staff 
members was the form data would take when it is compiled from a web-based survey. 
While there are generally standard methods to convert data from flat files or other forms 
to the data analysis file, there were concerns that the process would be time consuming, 
adding to the research analysts’ already burgeoning workload, and that it would increase 
the likelihood of corrupted data. Keeping these concerns in mind, a decision was made 
early in the process to test the new web survey software package put out by the same 
company that produces the data analysis software. This way, seamlessness would be 
paramount: while the survey is being developed the data file, with variable and value 
labels, is also being developed; and, the data that is collected from survey submissions 
would be automatically placed into the data file, alleviating concerns about conversion. 

Control. Another concern staff members had was the ability to control the process as 
well as the data. Actually, there was concern at two different levels. First, it was deemed 
important that the surveys stay on a campus-owned server, since there is sometimes 
proprietary data or sensitive data derived from the surveys conducted by the OIR. There 
was a fear that if the survey did not reside on a university server the owners of a private 
server - which are commonly used for different web survey packages - could collect ED’s 
or email addresses of respondents and use them for their own purposes or sell them. 

There was also concern over sensitive data, especially admissions-related, falling into the 
hands of competing institutions. 

Second, staff members thought it would be important that members of the OIR 
have direct control over the placement of the surveys on a server and extraction of the 
data once it is submitted. This was due to the fact that analysts are often under tight time 
frames within various stages of the survey process, and relying on a server administrator 
for such custodial actions may conflict with the need to place and replace surveys on the 
server in an instant. 
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Methodologically sound. Of course, the most important aspect of selecting a software 
package was its ability to allow the OIR to create and administer surveys that collect 
reliable data. One important element was to have the ability to create survey items in 
several different formats, based on the nature of the inquiry. The ability to combine good 
methodology with seamlessness and control in a software package would be the challenge 
of selecting the right product. 

Selection 

Early discussions with campus information technology professionals focused on 
the choice between developing our own program or purchasing a package. Creating a 
program had been done on other campuses using HTML for design and PERL or Cold 
Fusion for security and data collection, but there was no expertise in the OIR to complete 
such a task. After describing exactly what was needed, it was the consensus of the IT 
people that: 1) it would take a great deal of time and effort to build such a program, 2) 
they did not have the resources to support this type of initiative, and 3) that we should 
explore “off the shelf products.” 

Based on the OIR’s needs and expectations of a web-based software package, it 
was decided to test the web survey package put out by the same company that produces 
the office’s data analysis software. After consulting with the company’s sales 
representative concerning its attributes and viewing a live demonstration, it was 
determined that this product would make the process efficient and seamless. In keeping 
with the objectives of the project, it would allow the office to place it on a university 
server (but not control its custodianship since the OIR did not have its own server); and it 
was methodologically sound. The product was then forwarded to the university’s 
Information Technology Services (ITS) unit for testing. The web-based software package 
consisted of two primary components: the actual survey development software and its 
own server software, which needed to reside on a platform server with certain size 
attributes. It was ITS’s responsibility to test both the development software and server 
component for compatibility with the university server. This testing was extremely 
important since this was the first version of the product. ITS informed us that it was 
compatible with their servers and that they could support its use if we decided to purchase 
it. So we did. 



Implementation 

First Product 

It was decided that the first survey to be administered on the web by the OIR 
would be the 2000 Pre-Major Advising Survey, which is distributed to Sophomores in the 
Arts and Sciences and First-Year students in Engineering. These students are required to 
complete the survey in the spring semester in order to be able to register for fall courses. 
Thus, it was important to ensure that only those students who were eligible to complete 
the survey did, and that their survey completion was verified in order to release their 
registration hold. The server component was placed on the university server in the Fall of 
1999 and OIR staff began developing the actual web-based survey with the survey 
development component. 
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The server component, however, did not have an authentication function, so the 
first unexpected task was to build one to function with this product. Since the product 
selected was Java-based, an authentication function could not be built right into the 
server, and an electronic “wrapper” had to be built that could authenticate a user before 
accessing the survey. Since the OIR did not have access to the university server, staff 
members administering web surveys now had to rely on ITS professionals to build 
authentication programs for each survey (when needed) and load the survey on the server. 
The process was now getting more complicated and the critical path of survey 
administration was being extended, as shown in Figure 1. 

With an authentication program now written, the next step was to load and 
register the survey on the server. This also became an extended and iterative process. In 
order to receive data into a data file, a master file had to be registered on the university’s 

Figure 1. Critical path of web-based survey administration: Initial phase 




network server. There were several technical problems, however, relating the registration 
of the master file. Some of the problems were on the server end, and some were on the 
survey development end. By January of 2000 after several weeks of struggling 
with the first version of the product, a second version was released and the OIR was 
selected as a beta site. The second version was to have addressed many of the server 



interaction problems ITS was experiencing. While expectations were high that the 
process of loading and registering surveys would now be smoother, initial results were 
not positive. Compounding the difficulties was the departure of the ITS professional who 
had been assigned to the project. 

While the new ITS contact was a very good programmer, he needed time to come 
up to speed on the project. After experiencing more difficulties with registering the 
master file and loading the survey, a successful load and registration was finally 
completed. In March of 2000 a test survey was on the server and receiving data. Since 
sophomores and first-year engineers started registering for fall classes in late March, there 
was very little time to do extensive testing on the survey. Staff members and student 
research assistants in the OIR and ITS staff members submitted test data. Test data was 
collected and no serious problems reading the form or submitting responses were 
identified. Campus-wide testing was not done. 

When the survey went live and students were directed to complete the survey, 
many were unable to access and complete it. While there were some problems with 
identification numbers in the authentication process, the main problem identified was that 
students were unable to load the survey on their browsers or that it froze or crashed while 
they were completing it. The OIR then had to supply paper versions of the survey to 
those who could not load the survey. Of the 1,400 eligible students to complete the 
survey, a total of 866 submitted responses via the web, for a web response rate of 62%. 
Another 101 students completed paper surveys, for a combined response rate of 69%. 
Usually around 10% do not register for the fall of their Junior year due to study abroad 
plans, so there was still about 20% who did not complete the survey who still needed to 
register for classes. A determination was made by members of the registrar’s staff to 
allow students to register regardless of whether or not they had completed a survey. 

Thus, due to the difficulties in reading and loading the survey the OIR lost up to 27% of 
their potential web responses. 

Due to the loss of so many respondents, other web-based projects in the pipeline 
were put on hold until an evaluation of the pre-major advising experience could be 
conducted and procedures/modifications found to improve the process. When it was 
determined that a detailed training session on the software package - for both OIR staff 
and ITS staff - would be most appropriate, a two day session in the Fall of 2000 was 
scheduled with technical staff members of the company that produces the product. The 
goal of the training was to cover the survey process in a comprehensive manner, 
including survey development and server interaction. 

After the training was completed the OIR decided to proceed slowly in 
administrating web surveys. The Residential Life Survey, which is administered in 
January of each year to all students living in campus residence facilities, would be the 
next survey to go on-line, followed by an evaluative survey of the new student services 
center. The process of loading and registering the survey went smoother than the Pre- 
Major Advising survey. There was also less feedback concerning problems with loading 
and submitting the survey by respondents. But, even though incentives were offered for 
completing the surveys, there was no requirement to complete these surveys. Thus, as the 
OIR staff learned later, many students did have trouble loading the survey, but since it 



was not required, like the Pre-Major Advising survey, many simply disregarded it and did 
not make the effort to contact the OIR with problems. The final response rate for 
Residential Life was 34%, about 15% lower than previous paper administrations. The 
response rate for the student service center survey was, at 17% , lower than expected. 

To learn more about the lower response rates, a focus group was held with 
undergraduate students who had recently received emails announcing the residential life 
and student services center surveys. Most had not completed either. While they prefer to 
complete web surveys as opposed to paper surveys, they generally are not motivated to 
complete any survey unless it is compelling. They felt that a well-defined subject heading 
on the email announcing the survey is important, but are not necessarily motivated by 
prizes unless it is monetary in value. One issue that did not come out of the focus group 
was technical difficulty in loading and submitting the surveys 

In the meantime, however, as the Residential Life Survey was being administered 
and plans to do the Pre-Major Advising Survey (renamed Sophomore Survey) were being 
discussed, the OIR staff began to explore alternative packages for conducting web 
surveys. While progress was being made in utilizing the current software package, the 
OIR staff came to a consensus that there were three problems in using the product: 1) the 
survey development part of the program was somewhat labor intensive and clumsy; 2) the 
need to have ITS create wrappers and register each survey was not very efficient; and, 
most importantly, 3) there seemed to be no clear answer to resolving the problem of 
survey compatibility with a diversity of browsers on the respondents’ end. Multiple 
contacts with the company’s technical support did not provide an answer. The OIR 
became keenly aware of this problem when the Sophomore Survey was administered in 
March. When students called the OIR to report problems, they were asked what browser 
they were using and if they had enabled Java in the browser. Table 1 highlights the 
browser versions that generally worked and those that did not 1 . 

Table 1. Browser version compatibility with initial web surveying program 



Survey worked 


Survey did not work 


Netscape 4.5 


Netscape pre-4.5 


Netscape 4.7 


Netscape 6.0 


Exlorer 5 installed after 
October, 1999 


Explorer 5 installed before 
October, 1999 




Any Mac 



Second product 



1 These were general trends based on information supplied by the respondents. 
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With these three problems in mind, the OIR staff began collecting information 
from both campus-based IT consultants and professionals in the institutional research 
field with experience or knowledge of web surveying to find out how to address these 
problems. An analysis by a consultant at Tufts concluded that the only way to 
comprehensively address these problems was for the OIR to have its own server, which 
was not feasible due to the need for additional staff to run and maintain it.. 

Many alternative products were suggested by other IR professionals, and several 
were tested. The product that was selected for a trial run addressed all three of the 
necessary conditions cited by the OIR staff. First, it was very easy to use in the 
development stage. The question editor is very intuitive and display templates save 
considerable time in formatting. Second, the surveys reside on a private, non-university 
server but the user has complete control over the publishing and republishing of surveys 
to the server. No intermediary technical staff person is needed. The company also 
ensures, in writing, the confidentiality and security of the data. And third, while this 
product is also Java-based, there are no browser compatibility problems. Unfortunately, 
one goal the OIR did not reach in implementing this second product was seamlessness. 
While there are extra steps needed to convert the raw data to data that will reside in the 
standard data analysis software, those extra steps are well worth the effort based on the 
time and effort saved on the front end of the process. The critical path now involves less 
players and places more control in the hands of the OIR, as shown if Figure 2. 



Figure 2. Critical path of web-based survey administration: Second phase 
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Conclusions 



The process of selecting hardware and software to maximize survey effectiveness 
as well as to make the entire survey administration process efficient was a complex task. 
The institutional research office had three goals: 1) to implement a system that was as 
seamless as possible - that is, to integrate survey development, distribution, collection 
and analysis into an efficient process that minimized the critical path of survey 
administration, 2) to have institutional control over the process and resulting data, and 3) 
to implement a survey design and process that was methodologically sound. Analysis of 
the entire selection and implementation process shows that, based on the technical 
support and technical infrastructure available, seamlessness and control eventually had to 
be compromised in order to maintain methodological credibility. 



Implications 

The implications of this case study are great for those institutional researchers 
looking to migrate from paper surveys to web-based surveys. The lessons learned from 
this case study are numerous. It is critical to fully understand the level of technical 
support and complexity in your office and on your campus. In order to successfully 
implement a web-surveying system, the product chosen must meet your support needs. 
Technical support is critical. Make sure the technical support that is provided by the 
vendor is timely. Just because you do not hear about the problems it does not mean that 
they do not exist. Pilot testing was created for a reason. Invest time in exploring 
options. Be patient, go slow. Once you figure out how to best implement web-surveys 
on your campus, the results are astounding. 
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COLLEGE ELEMENTARY ALGEBRA STUDENTS PLACED BY 
ACCUPLACER SCORE, SCHOLASTIC APTITUDE TEST SCORE, OR 

PREREQUISITE COURSE 



Patricia Hirschy, Associate Professor of Mathematics 
Qing Mack, Director of Institutional Research 
Asnuntuck Community College 



Introduction 

Asnuntuck Community College is one of 12 colleges in the Connecticut 
Community College system. It is located in north central Connecticut, 25 miles north of 
Hartford, Connecticut, and 5 miles south of Springfield, Massachusetts. Asnuntuck has a 
service area of eight towns. In Fall 1999, 1,538 students were enrolled in more than 200 
credit courses at Asnuntuck. Ninety percent of the students were from the eight service 
towns, and 3% were from the neighboring state of Massachusetts. Asnuntuck is primarily 
a liberal arts college, and it offers 18 two-year degrees and 18 certificates. The college 
employs 24 full-time faculty and approximately 80 adjunct faculty. Sixty-one percent of 
the students are female, 24% are full time, 76% are part-time, and 68% are 22 years or 
older. 

MATH 101 (Algebra I) is an elementary algebra course offered by Asnuntuck for 
students who are not yet prepared to take a college level mathematics course. Some of 
these students have never had an algebra course, other students were not successful in 
previous algebra courses, and still others took algebra so long ago that they have forgotten 
their skills and need a review of the concepts. The purpose of this course is to offer 
students the opportunity to study or to review elementary algebra skills that are a 
foundation for college mathematics courses such as contemporary mathematics, 
elementary statistics, and intermediate algebra. 

Students are placed into Algebra I through one of three policies. One policy is 
based on the Accuplacer test score. The Accuplacer test is a computerized placement test 
with subtests in mathematics, English, and reading. Students who earn a high enough 
score on the arithmetic and algebra subtests can be placed directly into Algebra I. A 
second placement method is based on the Scholastic Aptitude Test (SAT). Students who 
score 400 or higher on the mathematics portion of the SAT can be placed directly into 
Algebra I without first taking the Accuplacer test. The third placement method is based 
on success in the prerequisite mathematics course. The prerequisite course for Algebra I 
is Prealgebra, a course with arithmetic and beginning algebra content. Students who have 
successfully completed Prealgebra or its equivalent with a grade of C or higher are 
eligible to enroll in Algebra I. 



Purpose of the Study 

The purpose of the study is to determine whether students placed by three 
different methods-Accuplacer score, SAT score, or prerequisite course-have different 
success rates in Algebra I. Comparing the success rates of students will provide an 
indication of the effectiveness of the placement methods, since they are established to 
identify those students whose mathematics background is appropriate for Algebra I. If 
the results of the study indicate that the student's mathematical skills are overestimated, 
then Algebra I will be at too high a level and the student will struggle to be successful. 

For this student, a better placement recommendation would be the prerequisite course. If 
the results of the study indicate that the student's mathematical skills are underestimated, 
then Algebra I will be at too low a level. The student will not be challenged in the course, 
and the resulting lack of motivation may affect the student's behavior and performance. 

In this case, a better placement recommendation for this student would be the next level 
mathematics course. 



Review of the Literature 

Students arriving at the community college door are often unprepared for college 
level classes (McCabe, 2000, p. 4). Forty-one percent need remediation in at least one of 
the basic disciplines: reading, writing, and mathematics. Twenty-five percent of the 
entering community college students have deficiencies in reading, and twenty percent 
have deficiencies in writing. Mathematics is the area with the greatest deficiency, with 34 
percent of community college students needing remediation. 

There are a number of reasons why students arrive at college without the 
necessary skills to do college-level work (McCabe, 2000, p. 39). In many states, high 
school exit competencies do not match entrance competencies for college. Also, high 
school students may be counseled into outdated general or occupational curricula. 

Further, mature students who begin college many years after high school graduation need 
courses to refresh their academic skills. A larger percent of high school graduates are 
continuing their education with post-secondary studies (Davis, 1989, p. 22). The 
increased demand for higher education resulted in a proliferation of open admissions 
colleges. The government has increased its financial support of developmental education, 
thus making it a more attractive offering. 

Along with the increased number of unprepared students came a need for 
placement policies to determine whether students had the skills to be successful in 
college-level courses. According to the American Mathematical Association of Two- 
Year Colleges (2001), the purpose of placement is to place students in accord with their 
educational goals and prerequisite knowledge (p. 1). The college policies should be 
applied equally to all student placements, and multiple measures should be included in 
the process. Colleges should validate the effectiveness of the placement policies and 
should evaluate them on an ongoing basis. 

Wattenbarger and McCleod (1989) investigated a variety of mathematics 
placement measures that have been used by community colleges. Statistical tests of the 
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relationship between student course grades and standardized college examinations such as 
the SAT and the ACT resulted in low correlations (p. 18). The SAT and ACT are 
intended as measures of general academic aptitude, so using them for placement decisions 
may be inappropriate (p. 20). Content-oriented tests would provide information that is 
more useful for placement decisions. Also, combining other factors such as high school 
mathematics background, student motivation, and student learning style would result in a 
more effective placement policy. 

Research indicates that mandatory testing and placement is an essential 
component of effective education (McCabe, 2000, pp. 42-45). It is important that 
colleges have information regarding the nature and extent of student academic needs so 
the appropriate support services can be developed and offered. Students enrolled in 
classes without the requisite skills are in danger of failing those courses. Instructors face 
students with widely differing skill levels, thus making it difficult to balance the needs of 
unprepared students with the needs of the course. 

In a review of ten exemplary college developmental education programs (McCabe & 
Day, 1998, p. 22), comprehensive placement and assessment efforts were critical factors of' 
success in every program. Bucks County Community College, for example, has established, 
mandatory placement and assessment policies to fulfill its goal to ensure that students "are. 
provided with an opportunity to begin their studies at an appropriate level" (p. 38). Prince 
George's Community College administers standardized testing to guarantee that students 
have a foundation for college-level instruction (pp. 79-83). As part of the process, the 
college offers a placement test confirmation procedure that includes a second departmental 
skills assessment and an intensive mathematics review course. 

Procedures 



Data Collection 

The study sample comprised Asnuntuck students who enrolled in all sections, day and 
evening, of Algebra I in Fall 1999 and who persisted sufficiently long to receive a course 
grade. Students who were "No Show" or who withdrew within the first two weeks of class 
were not included in the sample. An ex post facto research design was used, since the data 
was based on the records of students who have already enrolled in Algebra I and have 
received a grade. The information was retrieved from the Banner Student Record System, 
from current student folders, and from archived student folders. 

Demographic and academic information was collected on each student including gender, 
age, major, method of placement, Accuplacer score, SAT score, Prealgebra course grade, 
Algebra I course grade, number of credits completed prior to Fall 1999, and enrollment 
status (full-time or part-time). 

Data Analysis 

This research study applied both descriptive and inferential statistics. Descriptive 
statistics were used to organize and present the demographic information such as age, 
number of credits completed, and gender. Inferential statistics were used to determine if 
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there is a relationship between success in the course and placement methods. A Chi- 
square test was applied to determine if there was a statistically significant difference in 
success rates among students placed by Accuplacer score, prerequisite course, or by SAT 
score. The course grade was chosen as the measure of success since there are no 
standardized tests or departmental finals that would yield a different measure of student 
success. 



Results and discussion 

Of the 148 students who registered for the course, n = 107 students were eligible for 
the study. Forty-one students were not eligible due either to missing information or to 
placement other than by the three defined methods. A comparison of the study sample to the 
general college population showed that students who took Algebra I tended to be younger, 
enrolled full-time, in their first year of college, and more likely to be matriculated in a degree 
or certificate program (Table 1). The average age for all Asnuntuck students in Fall 1999 was 
32 years old, compared to 25 for those who took Algebra I. Fifty-three percent of the Algebra 
I students were fulltime, compared to 24% for the general college population. Eighty-nine 
percent of the Algebra I students were first-year students with 12 or fewer credits, compared 
to 75% of the general college population. Fifty-nine percent of the Algebra I students were 
matriculated, compared to 45% of the general college population. 



TABLE 1 

Demographic/ Academic Information 
Of 

Asnuntuck Students and Sample Population 





Total 


Full-Time 


Part-Time 


Male 


Female 


Under 22 


All College 


1538 


371 


1167 


598 


940 


490 


%of Total 


100% 


24% 


76% 


39% 


61% 


32% 


Ml 01 Students 


107 


57 


50 


41 


66 


60 


% of Ml 01 


100% 


53% 


47% 


38% 


62% 


56% 



1048 

68 % 

47 

44% 





Total 


1 st time 


Matriculatec 


1 st Year 


2nd Year 


All College 


1538 


498 


694 


1151 


387 


% of Total 


100% 


32% 


45% 


75% 


25% 


Ml 01 Students 


107 


44 


63 


95 


12 


% of Ml 01 


100% 


41% 


59% 


89% 


11% 



Students placed by the SAT score (n = 20) tended to be traditional age and enrolled in 
day sections. Students placed by the prerequisite course tended to be non-traditional age and 
enrolled in evening sections. Of the twelve second-year students, ten students were placed 
by the prerequisite course and two by Accuplacer score. 

The Chi-square test was first applied to the 107 students in the study. The 
independent variable was placement method-SAT score, prerequisite course, or 
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Accuplacer score. The dependent variable was success in course, where success is 
defined by C grade or higher. As can be seen from Table 2, there is no statistically 
significant difference in the data. Of note, though, is the range of success rates, from a 
low of 43.2% from prerequisite course placement to a high of 60% from SAT score 
placement. 



TABLE 2 

Ml 01 Attendees by all Three Placement Methods 





Ml 01 qrade by qroup 


Total 


C or better 


C- or lower 


Ml 01 W/SAT Count 

Attendees % within Ml 01 Attendees 


12 

60.0% 


8 

40.0% 


20 

100.0% 


Took Ml 00 Count 

% within Ml 01 Attendees 


16 

43.2% 


21 

56.8% 


37 

100.0% 


P/Ml 01 Count 

% within Ml 01 Attendees 


26 

52.0% 


24 

48.0% 


50 

100.0% 


Total Count 

% within Ml 01 Attendees 


54 

50.5% 


53 

49.5% 


107 

100.0% 



Chi-Square Tests 





Value 


df 


Asymp. Sig. 
(2-sided) 


Pearson Chi-Square 


1 .546 a 


2 


.462 


Likelihood Ratio 


1.554 


2 


.460 


Linear-by-Linear 

Association 


.084 


1 


.772 


N of Valid Cases 


107 







a - 0 cells (.0%) have expected count less than 5. The 
minimum expected count is 9.91 . 



The Chi-square test was then applied to student groups categorized by age, 
gender, course section (day or evening), and student status (first or second year). In some 
cases, the Chi-square test could not be performed due to low cell values. This difficulty 
arose for evening students placed by SAT score, for female students placed by SAT score, 
for traditional students placed by prerequisite course, for non-traditional students placed 
by SAT score, and for second-year students. For the remaining tests (listed below), there 
was no statistically significant difference at the 0.05 level. 

• All Algebra I sections 

• All day sections 

• Evening students placed by Accuplacer and prerequisite course 

• Female students placed by Accuplacer score and prerequisite course 

• Traditional age students placed by Accuplacer and SAT score 

• Non-traditional students placed by Accuplacer and prerequisite course 

• First-year students 



Even though there was not a statistical significance, there were two comparisons 
of note. For traditional students in day classes, those placed by SAT score had a much 
higher success rate of 62.5% compared to those placed by Accuplacer score with a 32% 
success rate. This resulted in a P-value of 0.055-higher than the 0.05 level of significance 
but close enough to warrant further investigation (Table 3). 

TABLE 3 



Traditional Age Students Enrolled in Day Classes 





Ml 01 qrade by qroup 


Total 


C or better 


C- or lower 


Ml 01 W/SAT Count 

Attendees % within Ml 01 Attendees 


10 

62.5% 


6 

37.5% 


16 

1 00.0% 


P/Ml 01 Count 

% within Ml 01 Attendees 


8 

32.0% 


17 

68.0% 


25 

100.0% 


Total Count 

% within Ml 01 Attendees 


18 

43.9% 


23 

56.1% 


41 

1 00.0% 



Chi-Square Tests 





Value 


df 


Asymp. Sig. 
(2-sided) 


Exact Sig. 
(2-sided) 


Exact Sig. 
(1 -sided) 


Pearson Chi-Square 


3.685 b 


1 


.055 






Continuity Correction* 


2.551 


1 


.110 






Likelihood Ratio 


3.713 


1 


.054 






Fisher's Exact Test 








.105 


.055 


Linear-by-Linear 












Association 


3.595 


1 


.058 






N of Valid Cases 


41 











a - Computed only for a 2x2 table 

b- 0 cells (.0%) have expected count less than 5. The minimum expected count is 7.02. 



Also, for students who were placed by prerequisite course (37 students), female 
students achieved a 51.9% success rate compared to the 20% success rate earned by male 
students. This result may be influenced by the fact that only ten male students were 
placed by the prerequisite course and only two passed Algebra I with a C grade or higher. 



Students Placed by Prerequisite Course 



Sex 


Ml 01 grade by g rou p 


Total 


C or better 


C- or lower 


Female 


Took M 100 


Count 


14 


13 


27 






% within useable 












sample by pic 


51.9% 


48.1% 


100.0% 


Male 


Took M100 


Count 


2 


8 


10 






% within useable 












sample by pic 


20.0% 


80.0% 


100.0% 



Conclusions and Recommendations 



The Chi-square test results of no statistical significance indicate that the three 
placement methods may be equally effective for the groups of students described in the 
Results and Discussion section. This shows that their chances of earning a C grade or 
higher in Algebra is not related to the method of placement. However, there are two 
groups of students for whom the analysis indicates a need for further investigation. 
Traditional students in day classes should be investigated to determine why students 
placed by SAT score were more successful than students who were placed by Accuplacer 
score (Table 3). Similarly, students placed by prerequisite course should be studied to 
determine why female students were more successful than male students (Table 4). In 
addition, this study was not able to confirm the effectiveness of placement methods for 
student groups with low sample sizes. 

Several recommendations can be made as a result of this study. Twenty-eight 
students who enrolled in the Fall 1999 Algebra I course were placed by alternative 
methods. These placements should be studied to identify patterns that led to student 
success and might be considered as additional placement procedures. The demographic 
and academic attributes of the students placed by SAT score should be studied to identify 
factors that could be considered in addition to the SAT score to better predict success in 
Algebra I. The curriculum of both Algebra I and Prealgebra should be reviewed to 
determine if there are curriculum changes that could increase the success rate for students 
who are placed by prerequisite course into Algebra I. Gender differences for this group, 
in particular, should be studied. 



Next Steps 

Analysis of the Fall 1999 Algebra I data for all 148 students should be continued 
to identify factors other than placement that might be used to predict success in Algebra I. 
Similarly, the data analysis should continue to identify factors that might flag students 
who are at-risk in Algebra I. Next, data should be collected that tracks the Fall 1999 
Algebra I students over the Spring 2000, Summer 2000, Fall 2000, Spring 2001, and Fall 
2001 terms to determine if further mathematics courses were taken, when the students 
took the courses, and how well the students performed compared to other students in 
these courses. Persistence rates and enrollment patterns (continuing enrollment, stop out, 
transfer, drop out, and graduation) should be included as part of the investigation. 
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Abstract 

Economic impact studies are utilized to demonstrate the beneficial effects that colleges and 
universities have on the local economy. In fall 1999, the University of Delaware conducted such a 
study to determine the economic impact that student, faculty and staff, and University 
expenditures have on the local community and state. This paper discusses the methodology and 
results of this study in detail. 



Introduction 

Economic impact studies are utilized by higher education institutions to demonstrate 
financial importance to their respective local communities. Stout suggests that colleges 
and universities exchange costs for benefits with their communities and economic impact 
studies help institutions to maximize their perceived financial benefits (1998). During 
fall 1999, the Office of Institutional Research and Planning at the University of Delaware 
(UD) conducted an Economic Impact Study. The purpose of this study was to determine 
the economic impact that student, faculty and staff, and University expenditures have on 
the local community and state. A survey was administered to the students and faculty and 
staff on the Newark campus, as well as local businesses in the Newark community. The 
final research report for this study contains three main sections that discuss the economic 
impact of University students, faculty and staff, and local businesses on Newark and the 
state of Delaware. 



Developing the Study 

A number of institutions have previously conducted economic impact studies (i.e., 
James Madison University, Pennsylvania State University, Southeastern Louisiana 
University, Universities of Arizona, Florida and Massachusetts-Amherst, Utah Valley 
State College, and Virginia Commonwealth University). These studies were reviewed 
before initiating the study at the University of Delaware. The study at UD borrowed the 
survey instruments utilized by Southeastern Louisiana University (Baldwin, Boeckman, 

& McKenzie, 1998). Modifications were made to these survey instruments to incorporate 
additional variables. A pilot test of the revised survey instruments was conducted among 
students and faculty and staff before the surveys were distributed to the actual sample 
population. In addition, conversations involved professors of economics at the University 
to ensure that the methodology and survey instruments were sound. One professor 
indicated a concern that while the survey instruments were comprehensive, people might 




-55 - 



53 



feel that they require too much effort to complete. However, individuals who participated 
in the pilot test did not comment on the lengthiness of the survey instruments. Comments 
and suggestions from the pilot test and various conversations were incorporated and the 
final survey instruments were developed. 

While the survey instruments would provide primary data on the direct expenditures of 
students and faculty and staff, it was also necessary to obtain secondary data for the 
University. The secondary data provided information on the direct expenditures and 
revenues of the University. The secondary data were obtained by contacting University 
departments such as conference services, dining services, athletics, special events, and 
purchasing. 

The method for this particular economic impact study is based on the Caffrey-Isaacs 
model developed for the American Council of Education (ACE) in 1971. This method, 
also known as the ACE method, is the most widely used to determine the economic 
impact of higher education institutions (Stokes & Coomes, 1998). It generates a series of 
impact indicators on the basis of simple linear cash-flow formulas. The first step of the 
ACE method is to identify the expenditures of the college community, including direct 
institution spending to vendors and local spending of students, faculty and staff, and 
visitors. The student expenditures should not include direct payments to the institution 
for tuition, housing, and food. A regional economic multiplier is then applied to the total 
impact of local spending by the institution, students, faculty and staff, and visitors. 

Another component of the ACE method is that the labor market impact is estimated by 
the increase of jobs in the area. It should be noted that Caffrey and Isaacs do not 
distinguish between the expenditure impacts of resident and non-resident students. The 
economic impact study developed at the University of Delaware utilizes many of the 
concepts of the ACE method. 



Methodology 

During the months of October and November 1999, the Economic Impact Study 
questionnaires were administered to students and faculty and staff on the Newark campus, 
as well as to local businesses. A follow-up mailing was conducted in February. The 
student questionnaire was administered to approximately 2,800 undergraduate and 
graduate students at the University. The students were randomly chosen to ensure a 
representative sample by class level, ethnicity, gender, campus status, residence status, 
and time status. The original student data set contained 688 surveys. The student 
response rate was approximately 25%. The final student data set was weighted during the 
analysis process to correctly represent the overall percentages of students by class level, 
gender, campus status, residence status, and time status. The weighted data thus provide 
findings from the sample of students to represent the Newark campus undergraduate 
population. » 

The faculty and staff questionnaire was administered to approximately 1,820 faculty 
and staff members on the Newark campus. The faculty and staff members were randomly 



chosen to ensure a representative sample by both employment and time status. The 
original faculty and staff data set contained 938 surveys. The faculty and staff response 
rate was approximately 52%. The final faculty data set was weighted during the analysis 
process to correctly represent the overall percentages of faculty and staff members by 
employment and time status. The weighted data thus provide findings from the sample of 
faculty and staff to represent the Newark campus faculty and staff population. 

The business questionnaire was administered to the business owners and managers of 
approximately 270 local businesses in the Newark area. The businesses contacted were 
located on Main Street and approximately a five-mile radius to the University. The 
business data set contained 90 surveys. The business response rate was approximately 
34%. 



Findings 



Student Economic Impact 

The mean monthly student income from all sources after taxes was approximately 
$1,020. The total mean monthly student expenditures spent in Delaware were 
approximately $780. Student expenditures ranged from housing to entertainment to 
medical and dental. Please note that students were asked to exclude University tuition, 
housing, and meal plans from their monthly expenditures. 

Student expenditures spent in Delaware varied by gender, time status, class level, 
residence status, and campus status. The total mean monthly expenditures spent in 
Delaware were approximately $860 for female students and $690 for male students. 
Female students spend a larger percentage of total monthly expenditures than their male 
peers on housing, utilities, telephone and cable, clothing, other retail, and medical and 
dental. Male students tend to spend more on entertainment and recreation, services, and 
automobiles. The percentage of total monthly expenditures for all other categories was 
equal. 

The total mean monthly expenditures spent in Delaware were approximately $520 for 
full-time students and $1,880 for part-time students. This large expenditure difference 
can be attributed to the fact that part-time students tend to be older and are most likely 
employed full-time. The percentage of total monthly expenditures that full-time students 
spend on telephone and cable, food and beverage, entertainment and recreation, clothing, 
and books and educational supplies is greater than their part-time peers. Part-time 
students tend to spend more on housing, utilities, services, other retail, automobiles, and 
medical and dental. 

The total mean monthly expenditures spent in Delaware by class level increased 
linearly. The largest expenditure category for all class levels except freshmen and 
sophomore students was housing. The housing expenditure for freshmen was extremely 
low due to the fact that 89% of freshmen live in University-approved housing. Freshmen 
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tend to spend the greatest percentage of their total monthly expenditures on food and 
beverage (20%) followed by books and educational supplies (17%). Sophomores tend 
spend the greatest percentage of their total monthly expenditures on automobiles (20%) 
followed by food and beverage (18%) and housing (17%). After housing expenditures, 
seniors, juniors, graduate students, and continuing education students tend to spend the 
greatest percentage of their total monthly expenditures on food and beverage and 
automobiles. 

The total mean monthly expenditures spent in Delaware were approximately $1,040 
for resident students and $490 for non-resident students. This large expenditure 
difference may be attributed to the fact that the total mean monthly expenditures are 
greatest for continuing education students and 79% of these students are Delaware 
residents. The percentage of total monthly expenditures that non-resident students spend 
on housing, telephone and cable, food and beverage, entertainment and recreation, 
clothing, and books and educational supplies was greater than their resident peers. 
Resident students tend to spend more on utilities, services, other retail, automobiles, and 
medical and dental. 

The total mean monthly expenditures spent in Delaware for on-campus students were 
approximately $250 and $1,100 for off-campus students. This large expenditure 
difference may be attributed to the fact that the on-campus students have minimal housing 
and utility expenditures. The percentage of total monthly expenditures that on-campus 
students spend on telephone and cable, food and beverage, entertainment and recreation, 
services, clothing, books and educational supplies, and other retail was greater than their 
off-campus peers. Off-campus students tend to spend more on housing, utilities, 
automobiles, and medical and dental. 

The estimated total annual expenditures spent in Delaware by the overall University 
student population were approximately $143,003,950' (see table 1 and chart 1). The 
breakdown of these annual expenditures is summarized on the following page. 



The annual expenditures for each category were calculated by multiplying the monthly expenditure by the 
student headcount for each term by the number of months in each term. The terms (number of months) 
included fall 1999 (4), winter (1), spring 2000 (4), and summer 1 and 2 (1.5 each). The total annual 
expenditures were the sum of these categories. 
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Table 1 

Annual Expenditures Spent in Delaware by Overall University Student Population 





Expenditures Per 
Year ($)' 


% of Total 
Annual 
Expenditures 


Housing 


44,506,332 


31.1 


Utilities 


7,113,717 


5.0 


Telephone and Cable 


6,748,91 1 


4.7 


Food and Beverage 


22,982,778 


16.1 


Entertainment and Recreation 


7,843,329 


5.5 


Services 


4,924,881 


3.4 


Clothing 


8,025,732 


5.6 


Books and Educational Supplies 


6,931,314 


4.8 


Other Retail 


8,208,135 


5.7 


Automobile 


20,793,942 


14.5 


Medical and Dental 


2,553,642 


1.8 


Other - 1 


1,641,627 


1.1 


Other - 2 


547,209 


0.4 


Other - 3 * 


182,403 


0.1 


Total Annual Expenditures 


143,003,952 


100 



Chart 1. Annual Expenditures 
Spent in Delaware by Overall University Student Population 
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In addition, students were asked to indicate up to six (6) Newark businesses that they 
frequent on a regular basis. Of the top 23 businesses that students mentioned, more than 
half (57%) were in the food and beverage industry, 30% were retail stores, 9% were 
grocery stores, and 4% provided general entertainment (i.e., video store). 

Faculty and Staff Economic Impact 

The total mean monthly faculty and staff household expenditures spent in Delaware 
were approximately $2,320. Faculty and staff expenditures ranged from housing to retail 
to education and tuition. Faculty and staff household expenditures spent in Delaware 
varied by employment status, residence status, and state of residence. Faculty members 
tend to spend the most in the state of Delaware, followed by professional staff, hourly 
staff, and salaried staff. The largest expenditure category for all employment status 
groups except hourly staff is housing. After housing expenditures, faculty, professional 
and salaried staff tend to spend the greatest percentage of their total monthly expenditures 
on food and beverage followed by automobile. The largest expenditure category for 
hourly staff is automobile. After automobile expenditures, hourly staff tend to spend the 
greatest percentage of their total monthly expenditures on housing followed by food and 
beverage. 

The total mean monthly expenditures spent in Delaware for resident faculty and staff 
members were approximately $2,690 and $1,120 for Delaware non-resident faculty and 
staff members. Delaware residents tend to spend the greatest percentage of their total 
monthly expenditures spent in Delaware on housing (32%) followed by food and 
beverage (16%) and automobile (14%). Delaware non-residents tend to spend the 
greatest percentage of their total monthly expenditures spent in Delaware on food and 
beverage (21%) followed by education and tuition (17%) and automobile (12%). 

The total mean monthly faculty and staff expenditures spent in Delaware varied by 
state of residence. Delaware residents spend the most in the state of Delaware, followed 
by Maryland, Pennsylvania, and New Jersey residents. Delaware residents tend to spend 
the greatest percentage of their total monthly expenditures spent in Delaware on housing 
(32%) followed by food and beverage (16%) and automobile (14%). Maryland residents 
tend to spend the greatest percentage of their total monthly expenditures spent in 
Delaware on education and tuition (20%) followed by food and beverage (19%). 
Pennsylvania residents tend to spend the greatest percentage of their total monthly 
expenditures spent in Delaware on food and beverage (27%) followed by automobile 
(16%) and education and tuition (13%). New Jersey residents tend to spend the greatest 
percentage of their total monthly expenditures spent in Delaware on other retail (30%) 
followed by automobile (29%) and food and beverage (23%). 

The estimated total annual expenditures spent in Delaware by the University Newark 
campus faculty and staff population were approximately $94,501,840 2 (see table 2 and 
chart 2). The breakdown of these annual expenditures is summarized below: 



2 



The faculty and staff annual 



expenditures were based on the Newark campus only (n=3,393). 
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Table 2 



Annual Expenditures Spent in Delaware by Overall University 
Faculty and Staff Population 





Expenditures Per 
Year ($) 2 


% of Total 
Annual 
Expenditures 


Housing 


28,094,040 


29.7 


Utilities 


6,066,684 


6.4 


Telephone and Cable 


3,012,984 


3.2 


Food and Beverage 


15,227,784 


16.1 


Automobile 


12,540,528 


13.3 


Medical and Dental 


3.420,144 


3.6 


Services 


3.745,872 


4.0 


Clothing 


3,705,156 


3.9 


Other Retail 


3,664,440 


3.9 


Entertainment and Recreation 


2,687,256 


2.8 


Education and T uition 


7,247,448 


7.7 


Other - 1 


3,745,872 


4.0 


Other - 2 


1,302,912 


1.4 


Other - 3 


40,716 


0.04 


Total Annual Expenditures 


94,501,836 


100 



Chart 2. Annual Expenditures 

Spent in Delaware by Overall University Faculty and Staff Population 



30.000. 000 

25.000. 000 

20 . 000 . 000 

15.000. 000 

10 . 000 . 000 
5,000,000 

0 




^ ^ 









N fl 



.sP 



<r /// /vVV 4 4 4 

/// / ^ V/V ° 

// / /y 






<0 






-f <f 



Expenditure Category 




-61 - 



65 




University Economic Impact on Local Businesses 



Local businesses indicated that they employ a number of current University students 
and alumni. They also indicated that they benefit from revenues generated by University 
students and faculty and staff. A number of the local business respondents indicated that 
the University and its community influenced decisions regarding products and services. 

In addition, the business respondents reported that the University and its community 
influenced decisions regarding scheduling events and sales. Other business decisions that 
are influenced by the University and its community included advertising, hiring, and 
business location choice. A number of the business respondents indicated that the 
University and its community were an asset to their business. In addition, the respondents 
indicated that the University had a positive influence on the Newark community and 
business sales. One respondent indicated that the University brought “vibrant” life to the 
community. Respondents also indicated that the University of Delaware made Newark 
“work” and the effect of the University and its community on their business as a whole 
was overall positive. Business respondents indicated that the faculty and staff at the 
University were a positive influence on their enterprise and that the University was 
important in making their operation successful, a driving force in the market share, 
provided competition, and encouraged a diverse population. 



Overall University Economic Impact 



University Revenues 

The University’s largest source of operating revenue was tuition and fees. In addition 
to operating revenue, the University generated revenue through special events and 
activities. For example, during the 1998-99 fiscal year, Clayton Hall hosted a number of 
meetings and events both internally and externally. Approximately 58% of the events 
hosted were external. The external meetings and events included the following types: 
corporate, associations, government, non-profit, public relations, religious, social, and 
education. In addition, during the 1998-99 fiscal year, 246,221 visitors attended the Bob 
Carpenter Center for intercollegiate athletic events, trade shows, and concerts. Eleven of 
the events at the Bob Carpenter Center consisted of two comedy shows, two children’s 
shows, one family show, one specialty show, and five concerts. Approximately 48,965 
visitors attended these 1 1 events and approximately $1,127,800 was generated in revenue. 

University Expenditures 

The University of Delaware is the 8th largest employer in the state of Delaware. 

During fall 1999, the University employed approximately 3,400 faculty and staff 
members on the Newark campus. The University compensated these employees 
approximately $154,775,980. 





The University makes a number of purchases through both Delaware and non- 
Delaware vendors. During the 1998-99 fiscal year, the University purchased 
approximately $62,835,400 worth of products and services through Delaware vendors. 
Purchasing in Delaware accounts for 41% of the University’s overall purchasing. 

Economic Impact Summary 

The direct expenditures of students, faculty and staff, and the University account for a 
large part of the University's economic impact on the state of Delaware. These direct 
expenditures lead to indirect purchases where additional services are purchased, 
employees are paid, and these employees, in turn, make additional expenditures. This is 
called the “multiplier” effect. The “multiplier effect” has also been defined as the ratio of 
increased income to increased spending (Stokes & Coomes, 1998). The total economic 
impact of the University of Delaware was calculated by applying a multiplier of 1.9 3 to 
the direct expenditures. The estimated total economic impact of student and faculty and 
staff direct expenditures and University purchasing is summarized below: 

Table 3 

Annual Expenditures Spent in Delaware by the University of Delaware and Its Community 





Estimated Spending 
in Delaware 
Per Year 


Overall Economic 
Impact 


Student Expenditures 


$143,003,952 


$271,707,509 


Faculty and Staff Expenditures 


$ 94,501,836 


$179,553,488 


University Purchases 


$ 62,835,388 


$119,387,237 


Total Economic Impact 


$300,341,176 


$570,648,234 



During 1999, the University and its community spent approximately $300 million in 
Delaware. These estimated expenditures spent in Delaware ($300 million) are 3 times the 
state operating appropriations level ($90 million). The estimated total economic impact 
of the University of Delaware is approximately $570 million. 

The University of Delaware is also responsible for generating additional jobs for 
businesses that provide goods and services to the University and its community. The 
Bureau of Economic Analysis suggests that approximately 36 jobs are generated for each 
additional $1 million dollars of output 3 . Looking at the estimated student and faculty and 
staff expenditures and University purchases in the state of Delaware, approximately 
10,810 new jobs are generated which increases the overall economic impact of the 
University of Delaware. 



3 U.S. Department of Commerce - Bureau of Economic Analysis. (1992). Regional multipliers: A user 
handbook for the regional input-output modeling system (RIMS 11). 3rd ed. Washington, D.C.: U.S. 
Government Printing Office. 
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Conclusion 



This study provides evidence that the University of Delaware’s impact on the local 
community and state through economic benefits is immense. The students and the faculty 
and staff contribute a great deal to the local and state economy through their personal and 
household expenditures. Local businesses indicate that the University and its community 
are a positive influence and contribute to their success. Overall, the return on the state’s 
investment in the University of Delaware is approximately 3 times greater than its initial 
investment. 

In addition to economic benefits, the University provides a number of additional 
benefits to the local community and the state of Delaware. These include, but are not 
limited to, employment opportunities, social and cultural events, educational 
opportunities, and community development. 

Conducting an economic impact study has provided the University with the necessary 
means to effectively communicate the value of the University in economic and social 
terms. This proves to be very useful when interacting with decision makers that include, 
but are not limited to, government officials, local businesses, and the local community. 
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Introduction 

The changing role of the institutional researcher: From statistics expert to people expert 

“Institutional researchers have long struggled with the definition of their 
profession, or indeed, whether what they do may be considered a profession” (Huntington 
& Claggett, 1992). From the office of statistics institutional researchers are moving 
toward more dynamic roles such as those related to planning and administrative decision- 
making (Banks and Colby, 1989). 

Notwithstanding the differences in scope, staff size and qualifications between 
institutional research offices in two and four-year colleges, we all still have to address 
reporting requirements and accountability issues pertaining to the competitiveness of the 
high education market place (Cyphers, 2001). More and more institutional researchers 
are actively participating in strategic planning processes and implementation, becoming 
involved in college-wide values assessment, mission, vision, goals, and action strategies 
determination. 



Purpose of this paper 

Within the framework described above, this paper intends to present the case of 
the Lehigh Carbon Community College, which has just gone through its fifth strategic 
planning effort, the first ever conducted by the director of institutional research and 
effectiveness. The experience described in this paper, the responsibility to conduct or at 
least actively participate in their institution’s strategic planning process, is becoming 
more and more common for institutional researchers. In this paper the author presents a 
five-step strategic plan she adapted and the process she developed for implementing it on 
a larger scale, i.e., that of a community college. 

Developing a Strategic Plan Tailored to a Community College 

The Lehigh Carbon Community College 

Lehigh Carbon Community College (LCCC) was bom Lehigh County Community 
College in 1966, sponsored by the Lehigh and Carbon counties school districts. In 1994, 
the name was officially changed to Lehigh Carbon Community College. Today LCCC 
has graduated almost 16,000 students, approximately 30% of the total enrollment during 
its 35 years of existence. LCCC has an average of 4,000 credit students per semester, and 



an average of 14,000 credit-free students per academic year, making its annual population 
an estimated crowd of 22,000 students. 

LCCC’s Office of Institutional Research has been in existence since 1988, 
initially to provide statistical information to internal and external constituencies, mainly 
its sponsoring High School Districts, the Pennsylvania Department of Education, and the 
federal Integrated Postsecondary Education Data System. Always in compliance with the 
Middle States Association of Colleges and Secondary Schools requirements for 
accreditation, LCCC has taken every step necessary to improve its effectiveness process. 
One of these steps is the involvement of the Office of Institutional Research in the 
institutional planning process. 

Planning and strategic planning efforts at LCCC 

Prior to 1983, strategic planning at LCCC was limited. The dean of instruction, 
who formed an agenda for curricular matters and grant applications, and the dean of 
business affairs, who focused on physical plant issues, did most of institutional planning. 
In December 1983 the president created a planning group called Horizons Council, 
consisting of faculty, administrators, classified staff and trustees. The Horizons purpose 
was to provide leadership and direction to the college’s planning effort through 
participation from all three of the personnel groups in addition to the trustees. Students 
joined the council in 1984. The Horizons Council developed a first five-year planning 
document, updated every six months and finalized annually. Progress toward the 
objectives was monitored throughout and recorded in a year-end report. A major problem 
with this format was that there was no formal structure for the college community to 
participate in the review of objectives developed by the Horizons Council. Key segments 
of the college, such as the Business office, academic areas, and the Office of Institutional 
research were underrepresented. Distribution of the progress reports generated by the 
Horizons Council was limited to the president’s council, deans, and directors. 

In 1991, the Board of Trustees decided that LCCC needed a more comprehensive 
strategic planning process that included input from external constituencies. A Strategic 
Planning Committee was formed and initiated the first major strategic plan undertaking at 
LCCC. Through in-depth personal interviews with representatives of all LCCC’s sectors, 
as well as leaders of local industries, businesses, professions, agencies and organizations, 
the college published in 1992 its LCCC 2000: A Strategic Vision for Lehigh County 
Community College. Despite the effort, many faculty and staff members felt that too 
much of that strategic plan content came from a small group of people and the overall 
feeling of ownership was limited. In 1993 LCCC’s new president replaced the Horizons 
Council with the college-wide Strategic Planning Committee, with the intention of 
revising the plan every two years through community interviews. 

In 1994, the coordination of the Strategic Planning Committee fell under the 
jurisdicition of the recently created position of Vice-President for Research, Planning, and 
Community-Based Education. As an effect of changes and re-structuring, the Strategic 
Planning Committee met only a few times over a period of two years, and the first update 
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of LCCC 2000 was not published until 1995: LCCC 2000 and Beyond. Although the 
document contained updated background information and some changes in format, the 
basic findings and strategic directions remained unchanged. Numerous efforts continued 
to be made by LCCC to conduct an effective and inclusive strategic planning process. 

In 1995 planning sessions included representatives of LCCC internal and external 
communities; and in 1996 and 1997 planning forums involving teams of faculty, staff, 
students and trustees were held to formulate priorities for the next two-year period. 
Overall LCCC planning process was structured in the following manner: members of the 
Board of Trustees participated in a weekend retreat at the beginning of every academic 
year to discuss plans and overall strategies and goals for the institution. The president 
then developed his personal goals and objectives based on LCCC 2000 and Beyond. The 
president’s goals and objectives were then distributed to the staff, and subsequently each 
department and each staff member would formulate its goals and objectives aligned with 
the president’s and college’s determined path. 

This formal process, however, did not promote cohesiveness within the college. 
The majority of faculty and staff felt confused about the college’s planning efforts since 
so many changes had occurred in the years before. Faculty in particular were unsure 
about their role in planning and whether their views were really being considered. Trying 
to resolve the confusion, in 1996 college administrators reorganized the standing 
committee structure, revamping some committees and adding new ones. One of the new 
committees created then was the Planning and Budgeting Committee, with 
representatives from faculty, administration, classified staff, and students, whose mission 
was to review and to make recommendations about matter related to the planning and 
budgeting processes. 

In 1998 LCCC started a new strategic planning effort, with the newly formed 
Strategic Planning Steering Committee. With representatives of all areas of the college, 
and led by an external consultant, the effort resulted in the establishment of the 1998- 
2003 Strategic Plan. The 1998 Plan covered five main areas of the institution: (1) 
programs and services, (2) systems and processes, (3) people, (4) finances, and (5) 
facilities. This plan endured until 1999-2000, when the president of the college initiated 
his retirement process. After a national search the new president, Mr. Donald W. Snyder, 
was hired in 2000. During the period from 1995 to 2000 the college experienced a sharp 
decrease in enrollment and retention, and one of the new president’s first measure was to 
revamp and upgrade LCCC efforts in recruiting and retaining students, as well as 
promoting internal cohesiveness by implementing a new extensive and inclusive strategic 
planning effort. For this effort, LCCC counted on its recently hired Director of 
Institutional Research, whose job title is now Director of Institutional Research and 
Effectiveness. 

Institutional Research and Effectiveness at LCCC 

LCCC Office of Institutional Research was vacant during the 1999-2000 period, 
being temporarily occupied by the Enrollment Director to fulfill state and federal 



reporting needs. In the fall of 2000 SCT-Banner was implemented college-wide as the 
new administrative software. As the new Director of Institutional Research and 
Effectiveness, the author came to the college in September of 2000, just over a month 
after the new system had been implemented. Still starting its SCT-Banner learning curve, 
the whole college staff was struggling to gain competence in the new system, and yet state 
and federal level reports needed completion. In the midst of learning the new job and the 
new system, the author still had pending the task of leading LCCC’s Institutional 
Effectiveness efforts, which would lead to the college’s compliance with the requirements 
of an upcoming Middle States Accreditation review. 

The model used at LCCC was based upon and adapted from the works of Bryson 
(1989), Lofquist (1990), Safrit (1990, 1991, 1992, and 1994), and Barry (1994). The 
five-steps model includes: 

1- S.W.O.T. profile - when the teams determine their strengths, weaknesses, 
opportunities and threats 

2. Determining organizational values - team members declare values that they embrace 
as individuals and, therefore, bring them to the team 
3- Establishing team's mission - team members define their team's mission by assessing 
how the team contributes to the whole organization, the team’s main reason for 
existing, the audiences the team serves, and the benefits resulting to the organization 
by virtue of the team members’ work 

4. Declaring the team's vision - the team establishes a vision, the highest standards the 
team wants to achieve as a group serving the large institution, by brainstorming where 
members think the team should go to serve the institution’s future needs, 

5- Determining goals, action strategies, and key performance indicators - team members 
decide practically and specifically what must be done to accomplish their mission and 
to reach their vision, as well as to determine how they get there and how they know 
that they have arrived. 

Implementing these five steps, however, took more than only five steps. During a 
period of six months a foundation for the work ahead was developed: (1) a teamwork 
handbook; (2) the institutional effectiveness committee defined and gathered; (3) networks of 
information and exchange with the college’s leadership team; (4) definition of teams in the 
midst of organizational re-structuring; (5) trainers trained; (6) activity days selected; and (7) 
infra-structure to support the project set in place. The process is not over yet, and to this date 
we still do not know how it is going to end. And that is the wonder of strategic planning. 



Forming a taskforce 



The Process 



It is very important to grant college-wide representation in the strategic planning 
taskforce, i.e., faculty, staff, classified staff, academic and institutional administration 
need to actively participate in this effort. At LCCC we called the strategic planning group 
the Institutional Effectiveness Task Force (IE Taskforce), and it included three faculty 





members, four administrative staff representatives, and three academic administration 
representatives. 

Selling the idea to the taskforce 

The college’s Leadership Team has appointed Taskforce participants. The Leadership 
Team is composed of the College’s president, vice-president, chief financial officer, and 
the deans. By virtue of being appointed and not consulted before, some IE Taskforce 
members were not necessarily happy to be there. Thus, the first responsibility of the 
Taskforce chair was to sell the idea to the other members of the taskforce. In the first 
LCCC IE Taskforce meeting, the Taskforce Chair brought materials to motivate and 
engage members in the topic, such as: 

• Articles and handouts on institutional effectiveness; 

• The Teamwork Handbook, a strategic planning model adapted for use by LCCC (see 
below); and 

• A summary of institutional effectiveness concepts as described by our accreditation 
agency, the Middle States Commission on Higher Education. 

The Taskforce Chair made herself available for questioning, listening to the expression of 
doubts and criticism, and granted everybody a good adaptation period. The LCCC 
Institutional Effectiveness Taskforce met every three weeks for four months. During this 
time, the fiercest opposition to the process came from the faculty members. They mainly 
expressed concerns that this process would not be effective, since is had been tried before 
without really changing the college. Some of this criticism focused on the process being 
"just another one," "an exercise in futility," "we have done this before," or "it's not going 
to work." It did not matter how many times the taskforce met and discussed the need for 
the process and the rationale behind it. Until the process had been completed, the faculty 
members of the taskforce were by far the toughest critics. 

Gaining the support of the College's Leadership Team 

Even though the process was initiated by the decision of the College's Leadership Team, 
it was still not clear to any of the parties, including the author’s, the extent to which we 
needed to work collaboratively, and especially how much the Leadership Team needed to 
support the initiative. When it became clear to the taskforce that it would be necessary to 
close the college in order to conduct the many teams self-assessment and examination, we 
needed the Leadership Team to schedule the events. In addition, we needed the authority 
of the Leadership Team to meet the college-wide resistance to the process. 

Defining the teams 

By virtue of the administrative re-structuring, the college was going through with 
a new president and renewed efforts to improve enrollment and retention, administrative 
and academic departments were in transition. Even the college's Leadership Team did 
not know yet what the final administrative format would be. After unsuccessful tentative 
efforts to divide the whole college into teams, and after waiting for more definitions from 
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the Leadership Team regarding the new administrative format, the IE Taskforce finally 
reached agreement. Faculty and staff were divided into academic areas and in budget 
areas, respectively. With the help and authority of the Leadership Team, the IE 
Taskforce conducted separate meetings with academic area heads and account directors, 
when the process was explained, and copies of the Teamwork Handbook were 
distributed. On those occasions the IE Taskforce Chair detailed the procedures, answered 
questions, and made herself available for addressing specific and personal concerns that 
might emerge. Academic heads and account directors then became responsible for 
distributing amongst their team members copies of the Teamwork Handbook, as well as 
explaining how it should be implemented before the D-Days scheduled for the whole 
college to conduct its strategic planning. We adopted the expression D-Days to refer to 
the days in which the College would be closed for the strategic planning to be conducted. 

It would be virtually impossible to close the whole college for one full day, so the 
IE Taskforce along with the college's Leadership Team decided to conduct the faculty and 
staff strategic planning processes on two different days. With summer approaching, it 
was becoming increasingly difficult to find a day in which the entire faculty would be 
able to participate. The Leadership Team decided that the afternoon of the Spring 
Graduation would be ideal for the Faculty D-Day, since traditionally all faculty 
participate in the graduation ceremony that would take place early that evening. The D- 
Day for staff was planned for a Friday, few weeks into the summer semester, when the 
movement of students was not so intense. 

Faculty was divided into 1 1 teams: 

1 . Healthcare professions 

2. Science 

3. Mathematics 

4. Human Services 

5. Technology 

6. Computer Science 

7. Business 

8. Counselors 

9. Humanities 

10. Social Sciences 

11. Learning Assistance 

Staff was divided into 19 teams: 

1. Academic Administration 

2. Academic Administrative Support 

3. Accounting Team 

4. Administrative Services (Administration, Human Resources, Safety and Security, 
Switchboard) 

5. Continuing Education Department 



6. Duplicating/ Mail/ Word Processing 

7. Educational Support: Learning Assistance Services 

8. Educational Support: Literacy and Job Training 

9. Information and Technology 

10. Institutional Advancement Team 

11. Institutional Effectiveness Team 

12. Learning Resource Center 

13. Marketing and Community Relations 

14. Operations and Maintenance 

15. Sites 

16. Student Accounts 

17. Student Life 

18. Student Services 

19. Workforce Training 

Training the trainers 

The LCCC faculty D-Day was scheduled for May, and the staff D-Day was scheduled for 
June 8th. Prior to those two major days, the Leadership Team and the Provost Team 
(composed by the former College’s provost and the deans) went through the strategic 
planning process. The objective was twofold: first to test the process, and second to 
"create" trainers who could help facilitate the May and June sessions. Also, the 
Institutional Effectiveness Task Force went through the process of addressing last 
concerns and generating trainers who understood the process and could help facilitate the 
sessions with faculty and staff. A tentative train-the-trainer manual was initiated but 
abandoned, as it proved relatively useless when compared to actually going through the 
process and clarifying questions on the spot. 

D-days and Infrastructure 

Closing the college two half-days took planning and collaboration among the 
Institutional Effectiveness Task Force, the Leadership Team, and the Human Resources 
Department. For both days, staff and faculty members were offered lunch by the College, 
coffee-break, and faculty members were served dinner right before the graduation, as many 
of them were staying at the college for the ceremony, with no time to go home between the 
two events. Signs were posted on the doors one week in advance to let students know that 
the college would be closed during specific periods. A memo, signed by the president, 
stressed the importance of participating and requested that those who would be unable to 
come should communicate in advance with the IE Taskforce Chair. Many staff, but only a 
few faculty members documented previous commitments; these parties were contacted to 
arrange subsequent meetings for update. 

Each faculty and staff member received a memo with the agenda for their 
respective days. This memo also contained the names of other teammates (many of these 
teams were still new due to the college’s re-structuring), the classroom to which they were 



assigned, and a reminder to bring their Teamwork Handbooks completed up to step four. 
The IE Task Force would have only four hours to conduct S.W.O.T. profiles, values, 
mission, and vision statements. Due to the limited time and impossibility of allocating 
more time to each day's activity, it was decided with the Leadership Team that the goals 
and action strategies would be completed individually by each team that could count on 
facilitators from the IE Task Force. 

Classes were selected close to one another to facilitate the event coordination; lists 
of names of team members were affixed outside the door of each assigned classroom. 
Classrooms were equipped with easels, easel pads, colored markers, and tape so the 
sheets could be taped into the walls as the work progressed. A few thoughts for the day 
were affixed in the walls for motivation and inspiration, as suggested by a consultant 
psychologist. The thoughts included references to the confidentiality of the discussions 
that would occur during the brainstorming; to the need for overcoming a complaining 
mode; and to obtain help for personal difficulties with the transition period the college 
was going through. These thoughts were titled Rules for the Day and were posted on 
signs on the classrooms walls. Despite the good intention, these signs generated some 
concern among a few faculty members because of being called "Rules." As a 
consequence of this reaction, the IE Taskforce changed the title to Thoughts for the Day 
for the staff D-Day. Thus no similar concerns arose during staff D-Day. For both events, 
facilitators gathered 30 minutes before the start of activities to re-group and clarify last 
minute questions and concerns. No further incidents occurred, and the IE Taskforce and 
the college's Leadership Team considered both events successful. 

Defining Goals and Action Strategies 

As decided, LCCC’s strategic planning process would follow the format bottom-up: top- 
down: bottom-up: and top-down again. The last piece of the first bottom-up part was still 
missing: the definition of goals and action strategies for each team. This piece would be 
important for the president and the Leadership Team's analysis of the college's direction 
for the next two-year budget period. The deadline for this analysis to start was the 
beginning of the Fall semester, therefore it was determined that each team should meet in 
their own time to go through the last step of this first phase of the process. The IE 
Taskforce and some account directors who have already been through the process were 
made available to facilitate the staff teams' processes. Since the majority of LCCC 
faculty is off during the Summer, it was determined that during Convocation Day, before 
the official start of Fall classes, academic teams would have a three-hour period to 
determined their goals. Once more IE Task Force members and other staff members 
already trained in the process would be available that day to facilitate. 
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Compilation 



As a result of LCCC’s college -wide strategic planning process, several compiling 
documents were produced, with more to be produced in the future: 

1. A summary of overall strengths, weaknesses, opportunities, threats, values, 
missions and visions of all teams for the president’s analysis 

2. A compilation of each teams work for the president’s analysis 

3. A compilation of trends of strengths and weaknesses for the president’s analysis 

4. A compilation of trends of opportunities and threats for the president’s analysis 

5. A summary to be distributed to the whole faculty and staff body of each team’s 
mission, vision, and goals after the final input from the president and the 
Leadership Team (in process). 

Lessons Learned 

1. There is no “by-the-book” way to conduct a college-wide strategic planning in the 
midst of concurrent organizational change 

2. Plan, plan, plan, plan, plan, plan, plan, plan ahead (if you can) 

3. Include everybody in the planning as much as you can 

4. Be very prepared for resistance - RESISTANCE IS NATURAL 

5. Be prepared to act as a counselor, psychologist, confidant and pacifier 

6. Assure and assure and assure everybody that everything will be all right in the end 

7. Each organization has its characteristics and personality 

8. Try to have advice from someone with some experience or who did it before to 
double check steps 

9. Be prepared to improvise and adapt to specific and unexpected circumstances 

10. Accept that the process will fall out of your hands at some moments and come 
back to you later, probably corrupted and changed 

11. Be persistent, be patient 



Positive Aspects of Model 

1 . I knew how to do it 

2. I worked in many other circumstances 

3. Promotes college-wide participation 

4. In the back-and-forth movement, it gives opportunities for revisions 



Negative Aspects of Model 

1. May take too long in a larger organization 

2. Gives opportunity for a lot of criticism, whining, and errors 

3. May get out of hand due to the need to use several facilitators 
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THE IMPACT OF A SERIES OF WRITING INTENSIVE COURSES 
ON SUCCESS ON THE WRITING PROFICIENCY REQUIREMENT 



Kevin B. Murphy 
Research Analyst 

Office of Institutional Research and Policy Studies 
University of Massachusetts Boston 



Introduction 

In April 1999, the Office of Institutional Research and Policy (OIRP) received a 
request from the Writing Proficiency Requirement (WPR) Committee for research on the 
connection between the curriculum and success on the WPR. The requirement consists of 
the successful completion of a timed essay examination, or the submission of a portfolio 
of work that includes several examples of papers written for courses and a new paper 
based on assigned readings and specific questions. It is designed to "...assist students in 
acquiring critical skills. Foremost among these is the ability to present ideas clearly, 
correctly, and persuasively in English prose" (UMB Undergraduate Catalog). The 
requirement must be successfully completed as a prerequisite for graduation from the 
College of Arts and Sciences (CAS) and from the College of Nursing (CN). It is a high 
stakes requirement. There is no alternative path to graduation. 

One focus was to be on a group of courses that were designed to prepare students 
for the WPR. These were the Core or “C” courses, which were offered in a number of 
disciplines throughout CAS. They were overseen by the Core Curriculum Office, which is 
also responsible for the administration of the WPR. The requirement called for students 
to complete five "C” courses. Of the five required core courses, three were to be at the 
100 level, and the other two at the 200 level. A core course emphasized the nature of 
knowledge and the methods of investigation that characterize the disciplines within its 
distribution area. Core courses provided instruction and practice in such intellectual skills 
and habits of thought as analytical writing, critical thinking, quantitative reasoning, and 
research techniques. In general, students were to complete their core courses before they 
attempted the WPR. Transfer students with 30 or more credits were not required to fulfill 
the requirement. This system was changed in fall 2000. The new First Year Seminar 
(FYS) courses replaced the old “C” courses for newly matriculating students. Students 
who matriculated prior to that date who were subject to the old rule are now required to 
complete a total of two “C” courses at either the 100 or 200 level. 

Before I could examine the relationship between the curriculum and the WPR, I 
needed to identify all of the steps leading up to the WPR. Therefore, I conducted a 
process evaluation. The results of this evaluation were presented in a previous report. 
However, a short summary would be valuable. 
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The Writing Proficiency Requirement should be viewed as a process that begins before a 
single course is ever taken at UMB, rather than as an event. There are well-established 
rules for the process. It was fairly easy to identify how the process works, or, at least, how 
it is supposed to work. It involved attending orientation, completing an English 
Placement Assessment (EPA) and receiving a placement recommendation, completing 
the recommended courses and the five “C” Courses, and attempting the WPR at about 60 
credits. If the student was successful, there were no further related requirements. If the 
student was unsuccessful, there were two specialized courses to prepare the students to 
retake the exam, and tutoring would also be made available. 

In practice however, we found that large numbers of students failed to attend 
orientation or to get an EPA recommendation, failed to complete the appropriate courses 
if they had a recommendation, and especially failed to complete all of the “C” courses 
that were designed to prepare them for the WPR. Because all freshmen were subject to 
the rule, this report will focus on the behavior of students who entered UMB as freshmen. 



Of the nearly 1,000 freshmen in the study, barely 17% utilized the system in the 
manner in which it was designed as may be seen in Table 1. 



Table 1: Freshman Compliance with the WPR Preparatory Process 



Group 


Number of Students 


Percent 


All first time freshmen 


993 


100.0% 


Freshmen with an EPA 


638 


64.2% 


Freshmen who complied with the EPA 


595 


59.9% 


Freshmen who complied with the EPA and 
completed at least 5 “C” courses 


170 


17.1% 



Analyzing the Impact of the “C” Courses 

Given the poor level of compliance with the system, the question of whether 
completion of C courses is important remained. This analysis will examine the impact 
of “C” courses for all freshmen, regardless of whether they had an EPA recommendation 
or whether they complied, if they had one. Significance tests were run on the differences 
between the number of “C” courses completed by students by whether they had an EPA 
recommendation, whether they complied with that recommendation, and whether they 
had one and complied with it versus those who either did not have an EPA or had one and 
did not comply with it. The differences were not significant in any case, and are presented 
in Table 2. 



Table 2: Significance Test Results on “C” Courses by EPA Status 



Comparison Groups 


Mean 


Difference 


|T| 

Value 


Probability >|T| 


EPA 


3.152 


.008 


.0683 


94.55% 


No EPA 


3.144 












EPA and complied 


3.149 


.08 


.2856 


77.53% 


EPA Did not comply 


3.069 












EPA and Complied 


3.149 


.006 


.0562 


95.52% 


No EPA /did not comply 


3.143 



Given these small differences, it seems reasonable to examine the impact of the 
number of “C” courses alone without considering EPA compliance status. 

The number of "C" courses taken before attempting the WPR varied considerably, 
with less than 30% of the students having completed at least five of the “C” courses prior to 
attempting the WPR. The specifics are presented in Table 3, which follows. 



Table 3: “C” Courses completed by Freshmen Prior to the First WPR Attempt 



‘C” Courses Completed 


Number of Students 


Percent of Students 


0 


101 


10.2% 


1 


101 


10.2% 


2 


156 


15.7% 


3 


158 


15.9% 


4 


198 


19.9% 


5 


235 


23.7% 


6 


37 


3.7% 


7 


7 


0.7% 



Methods 

The data used for this study come from official University of Massachusetts Boston 
files. Prior to June of 1996, only data for the most recent attempt was maintained on the 
database. Therefore, we would were unable to tell when courses were completed in 
relation to the first attempt if more than one attempt was needed. The system was 
changed for the June 1996 examination. Therefore, the analysis group is limited to 
students (freshmen in this study) who attempted the WPR for the first time between June 
1996 and June 2000 inclusive. Because of the attendance patterns of UMB students, this 
group included students who entered UMass Boston as freshmen as early as fall 1984 and 
as late as fall 1999. 





No attempt was made to account for the quality of the students’ “C” course 
experiences. This study is not intended to be an evaluation of the program as a whole. 

The analysis does not deal with assessment of program activities or of classroom 
implementation of those activities (Hughes). It simply focuses on the relationship 
between the students’ performance on the WPR and completing some number of “C” 
courses. Any final grade that carried credit was counted as the successful completion of 
the course. No attempt was made to control for the discipline in which the course was 
offered. For the purposes of this study, all “C” courses were created equal. 

Comparison of means tests and simple bivariate logistic regression models will be 
used. The dependent variable will be the result of pass (1) or fail (0) on the first attempt on 
the WPR. The independent variable will be the number of “C” courses successfully 
completed by the student prior to that first attempt. The overall first attempt pass rate was 
76.7%. Students who completed six or seven “C” courses will be folded into a category of 
five or more completed “C” courses. 

The analysis has been conducted using Stata for Windows® Version 6.0 

Results 

Even UMB students who entered as freshmen are not a homogenous 
group. About 58% of the students in this study were female, 36% were over age 25 
(including several who were over age 60), and the group is racially and ethnically diverse 
with significant numbers of international students. They came to the university with 
varying levels of preparation. Many are non-native English speakers. Nevertheless, the 
first step was to examine the relationship between the “C” courses and success for the 
group as a whole. The observed values for all freshmen entrants are presented in the 
following table. 



Table 4: Observed Pass Rates for AH Freshmen by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


72.3% 


75.3% 


72.4% 


75.3% 


76.8% 


82.1% 


N Size 


101 


101 


156 


158 


198 


279 



No particularly strong pattern is observed except that the largest jump in pass rates 
is for those who completed four courses versus those who fully complied with the 
program by completing at least the five recommended courses. Based on this, a new 
dummy variable called “full_c” was created that had a value of 0 if the student completed 
four or fewer courses and 1 if the student completed five or more. A comparison of means 
test was run on the pass rate variable by the full_c variable. Those who did not complete 
the full sequence (N=714) passed at a rate of 76.4%, while those who did (N=279) had a 
pass rate of 82.1%. The difference of just over 7% returned a T-statistic with an absolute 





value of 2.4957, which was significant at above the 95% level (98.73%). 

The more interesting question is whether the “C” courses have an incremental effect 
rather than being successful for only those who complete the program. Here a bivariate 
logistic regression was run. The results are presented in Table 5 below. The overall model 
returned a chi square statistic of 5.55. The probability of a larger chi square statistic by 
chance alone is about 1.85%. 



Table 5: Logit Estimates of Pass 1 st by Number of “C” Courses 



Pass 1st 


Coefficient 


Standard Error 


Z 


P>Z 


95% Confidence Interval 


“C”Courses 


.1007818 


.0427616 


2.357 


0.018 


.0169706 - .184593 


Constant 


.8846124 


.1481727 


5.97 


0.000 


.5941993 - 1.175025 



Stata allows one to predict the values of the dependent variable by variation in the 
independent variables. In Table 6 which reports those predicted values, we can see that, 
on average, each “C” course successfully completed prior to the first attempt at the WPR 
increases the probability of passing by about 2%. 



Table 6: Predicted Pass Rates for All Freshmen by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


70.8% 


72.8% 


74.8% 


76.6% 


78.4% 


80.3% 



While small, this 2% increment in the pass rate for all freshmen has some 
importance. If all of the students had completed the full program, we would predict that 
about 35 more of them would have passed the test on the first attempt. This would reduce 
the expenses associated with a second (or subsequent) WPR administration, of operating 
the two specialized support courses, and of the tutoring and other administrative supports 
that is offered to support those who have failed the WPR. 

Given the diversity of our student body, it is possible that completing “C” courses 
has more of an effect on some groups than on others. The first group I wanted to examine 
were the students who entered through the DSP program. The Directions for Student 
Potential (DSP) program is a free six-week, pre-admission summer program that provides 
academic advising, career planning, and personal counseling. DSP offers intensive 
workshops in reading, writing, mathematics, and study skills. Those students who 
successfully complete the program are admitted to the College of Arts and Sciences in the 
fall. It is designed for those students who show academic promise, but do not meet the 
traditional admission criteria. 

The mean number of “C” course completed was not significantly different for 
DSP and non-DSP students. The observed pass rate by number of “C” courses is 
presented in Table 7, which follows. 
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Table 7: Observed Pass Rates for DSP Entrants by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


35.7% 


48.4% 


59.4% 


62.5% 


65.1% 


75.5% 


N Size 


14 


31 


32 


24 


43 


53 



In this case, the strength of the pattern is immediately noticeable. A comparison of 
means test was run on the pass rate variable by the full_c variable. Those who did not 
complete the full sequence (N=144) passed at a rate of 56.9%, while those who did (N=53), 
had a pass rate of 75.5%. The difference of just over 18.5% has an associated T-statistic 
with an absolute value of 2.3974, which was significant at above the 95% level (98.25%). 

The logistic regression was then run. The results are presented in Table 8, which 
follows. The overall model returned a chi square statistic of 10.08. The probability of a 
larger chi square statistic by chance alone is about 0.15%. Only the number of “C” 
courses was significant. 



Table 8: Logit Estimates of Pass 1 st by Number of “C” Courses for DSP Students 



Pass 1st 


Coefficient 


Standard Error 


Z 


P>Z 


95% Confidence Interval 


“C’Courses 


.2730294 


.0880809 


3.100 


0.002 


.1003941 - .4456648 


Constant 


-.341596 


.3000381 


-1.139 


0.255 


-.9296598 -.2464678 



Once again, the predicted pass rates were obtained, and are reported in Table 9. 



Table 9: Predicted Pass Rates for DSP Entrants by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


41.5% 


48.3% 


55.1% 


61.7% 


67.9% 


74.6% 



Using the predicted values of both of the previous regressions and all other things being 
equal, the DSP students who take no “C” courses can expect their pass rates to trail those 
of the overall group who took no “C” courses by about 29.3%. However, those DSP 
students who complete the full sequence would expect their pass rates to lag those of all 
students who complete the full sequence by only 5.7% and to lag the overall group 
average by just over 2%. These differences suggest that completing the full sequence of 
“C” courses was particularly important for students who entered through the DSP 
program. 

Part of this may have been because of native language status. In this analysis, we 
used two separate indicators for native language status. One is “ESL”. While some people 
object to the term “ESL”, here it has a specific meaning. ESL students are students whose 
English language skills needed enough additional work that their EPA recommendation 
was for a sequence of English as a Second Language courses, or who did not have such a 
recommendation but took ESL courses anyway. Overall, about 10% (N=100) of all 





freshmen were ESL. 



The other language group was the non-native English speaking students. I will 
follow the practice of Hamp-Lyons (1996) and use NNS for non-native speakers and NS 
for those who are native speakers of English. This group is very difficult for us to 
identify because we have no flag for it in our computer systems. Non-native English 
speakers were identified by an ESL recommendation or course, the presence of a score for 
an ESL assessment, the presence of a TOEFL score, or a recommendation for one of the 
English composition courses designed specifically for non-native English speakers. While 
all ESL students were non-native English speakers, many non-ESL students were NNS. 
Overall, about 19.8% (N=197) of all freshmen were NNS. We understand this to be a 
serious undercount. Results from the 2000 administration of the National Survey of 
Student Engagement and from two surveys administered in fall 2000 and spring 2001 in 
UMB First Year Seminar (FYS) courses indicate that the true percentage of NNS in our 
student population is probably closer to 40%. Separate regressions were run for the NNS 
students and for the ESL subset of NNS. 

A comparison of means test was run on the number of “C” courses completed by 
NNS vs. NS students. On average, NS students completed 3.3 courses while NNS 
students completed only 2.7 courses. This difference of about .6 of a course had an 
associated T-statistic with an absolute value of 4.5955, which falls significantly above the 
99% confidence level. This difference becomes important if there is a positive 
relationship between completing the courses and success for NNS students. 



Table 10: Observed Pass Rates for NNS students by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


50.0% 


55.9% 


59.1% 


63.3% 


66.7% 


81.3% 


N Size 


34 


34 


44 


49 


42 


48 



In this case, the strength of the pattern is also immediately noticeable. A comparison 
of means was run on the pass rate variable by the full_c variable. Those NNS students 
who did not complete the full sequence (N=203) passed at a rate of 59.6%, while those 
who did (N=48) had a pass rate of 81.3%. The difference of over 21.5% has an associated 
T-statistic with an absolute value of 2.8389, which was significant at above the 99% level 
(99.51%). 

The logistic regression was then run. The results are presented in Table 1 1 below. 
The overall model returned a chi square statistic of 9.67. The probability of a larger chi 
square statistic by chance alone is about 0.19%. Only the number of “C” courses was 
significant. 

Table 11: Logit Estimates of Pass 1 st by Number of “C” Courses for NNS Students 



Pass 1st 


Coefficient 


Standard Error 


Z 


P>Z 


95% Confidence Interval 


“C’Courses 


.2453326 


.0803607 


3.053 


0.002 


.0878284 - .4028367 


Constant 


-.0782742 


.2433763 


-0.322 


0.748 


-.5552829 - .3987346 
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so 



Once again, the predicted pass rates were obtained, and are reported in Table 12. 

Table 12: Predicted Pass Rates for NNS Students by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


48.0% 


54.2% 


60.2% 


65.9% 


71.2% 


76.3% 



Using the predicted values of the previous regressions, the NNS students who take no 
“C” courses can expect their pass rates to trail those of the overall group who took no “C” 
courses by about 22.8%. However, those DSP students who complete the full sequence 
would expect their pass rates to lag those of all students who complete the full sequence 
by only 4% and to lag the overall group average of 76.7% by just 0.4%. While these 
differences are statistically significant, they suggest that completing the “C” courses was 
also particularly important for NNS. 

A comparison of means test was run on the number of “C” courses completed by ESL vs. 
non-ESL students. On average, non-ESL students completed 3.2 courses while ESL 
students completed only 2.4 courses. This difference of about .8 of a course had an 
associated T-statistic with an absolute value of 4.8008, which falls significantly above the 
99% confidence level. 



Table 13: Observed Pass Rates for ESL Students by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


Pass Rate 


40.0% 


36.8% 


45.5% 


53.3% 


64.7% 


75.0% 


N Size 


15 


19 


22 


15 


17 


12 



In this case, the strength of the pattern is also immediately noticeable. A comparison 
of means was run on the pass rate variable by the full_c variable. Those ESL students 
who did not complete the full sequence (N=88) passed at a rate of 47.7%, while those 
who did (N=12) had a pass rate of 75%. The difference of over 27% has an associated T- 
statistic with an absolute value of 1.7833, which was not statistically significant 
(P > |T|=92.24%). Because of the very small numbers, statistical significance is hard to 
attain, but the numbers still bear reporting. It is troubling that only 12% of the ESL 
students completed the full sequence of “C” courses. This is significantly below the 
overall mean of 28.1%. Although the NNS students also have a mean significantly below 
the group mean, when the ESL students are eliminated from that group, the difference is 
no longer statistically significant. 

The logistic regression was then run. The results are presented in Table 14 below. 
The overall model returned a chi square statistic of 6.13. The probability of a larger chi 
square statistic by chance alone is about 1.33%. Only the number of “C” courses was 
significant, but the constant is very close and the number of observations is small. 
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Table 14: Logit Estimates of Pass 1 st by Number of “C” Courses for ESL Students 



Pass 1st 


Coefficient 


Standard Error 


Z 


P>Z 


95% Confidence Interval 


“C’Courses 


.3181964 


.1324398 


2.403 


0.016 


.0586192 - .5777737 


Constant 


-.7069913 


.3713679 


-1.904 


0.057 


-1.434859 - .0208765 



Once again, the predicted pass rates were obtained, and are reported in Table 15. 



Table 15: Predicted Pass Rates for ESL Students by Number of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 


Pass Rate 


33.0% 


40.4% 


48.2% 


56.2% 


63.8% 


70.7% 



Each course completed raises the probability of passing by about 7%. Using the 
predicted values of the previous regressions, the ESL students who take no “C” courses 
can expect their pass rates to trail those of the overall group who took no “C” courses by 
about 37.8%. However, those ESL students who complete the full sequence would expect 
their pass rates to lag those of all students who complete the full sequence by only 9.6% 
and to lag the overall group average of 76.7% by just 6%. While these differences are 
statistically significant, they suggest that completing the “C” courses was also particularly 
important for ESL students. 

The next difference we wanted to test was for academic preparation. We used the 
Verbal SAT as a proxy. Many of our students are exempt from submitting SAT scores. 
Just over 63.5% (N=631) of all freshmen submitted SAT scores including 77.2% of the 
DSP students, 60% of the ESL students, and 54.2% of the NNS students. 451 of the 631 
students who submitted VS AT scores scored below 500. Our first step was to look at the 
pass rates by VSAT score. To do this we set up 100 point groupings which are reported in 
Table 16. 



Table 16: Observed Pass Rates by VSAT Score 



VSAT 


200-299 


300-399 


400-499 


500-599 


600-699 


700 and Above 


Pass Rate 


51.3% 


69.9% 


83.4% 


86.7% 


92.6% 


100% 


N Size 


115 


143 


193 


120 


54 


6 



It seems obvious that whatever the shortcomings of the VSAT, it has fairly strong 
predictive power for success on the WPR. 

I set an arbitrary cutoff of 500 to test for impacts on these students. Among other 
uses, a score of above 500 on the VSAT exempts incoming freshmen from certain entrance 
assessment testing mandated by the Massachusetts Board of Higher Education. I then 
conducted significance tests on the mean number of “C” courses completed by VSAT level. 
For both of the groups, the students who failed completed about .4 of a course less than those 
who passed on the first attempt, which was statistically significant for the below 500 group 



(|T|=2.3736) but with no statistical significance for the above 500 group. The mean number 
of completed “C” courses was the same for the two groups. 



Table 17: Observed Pass Rates for Students with VSAT Scores by VSAT Level and 
Number of “C” Course 



Courses Completed 


0 


1 


2 


3 


4 


5 or More 


>500 Pass Rate 


91.7% 


83.3% 


85.3% 


83.3% 


90.0% 


94.0% 


> 500 N Size 


24 


12 


34 


30 


27 


50 


<500 Pass Rate 


59.1% 


72.4% 


64.9% 


66.7% 


75.3% 


77.4% 


< 500 N Size 


44 


58 


74 


63 


97 


115 



Here we see that regardless of the number of “C” courses completed, the mean pass rate 
for students with above 500 scores is above that of the students with below 500 scores, and 
above the mean for the overall group. 

Logistic regressions were run for the group as a whole and for those above and below 
500. The results were significant for the group as a whole, but virtually all of the power is for 
the students below 500. For the group with 500 or above scores, there is no statistically 
significant relationship between the number of “C” course and passing the WPR on the first 
attempt. However, for the group who scored below 500, the relationship was significant. The 
results are presented in Table 18 below. The overall model returned a chi square statistic of 
5.59. The probability of a larger chi square statistic by chance alone is about 1.81%. Both the 
number of “C” courses and the constant were significant. 



Table 18: Logit Estimates of Pass 1 st by Number of “C” Courses for Students with 

Sub-500 VSAT Scores 



Pass 1st 


Coefficient 


Standard Error 


Z 


P>Z 


95% Confidence Interval 


“C’Courses 


.1384777 


.0588701 


2.352 


0.019 


.0230945 - .253861 


Constant 


.4791467 


..2001355 


2.394 


0.017 


.0868885 - .871405 



Once again, the predicted pass rates were obtained, and are reported in Table 19. 



Table 19: Predicted Pass Rates for Students with Sub-500 VSAT Scores by Number 

Of “C” Courses 



Courses Completed 


0 


1 


2 


3 


4 


5 


Pass Rate 


67.4% 


69.6% 


71.7% 


73.7% 


75.6% 


77.8% 



Each course completed raises the probability of passing by about 2%. Completion of 
the full sequence of courses raises the probability of passing on the first attempt to above 
that of the overall group. 
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Conclusions 



The “C” courses were designed to teach a number of skills and to prepare the 
student for the Writing Proficiency Requirement. Because most students did not fully 
comply with the “C” course requirement, the variation in the number of courses 
completed allowed us to assess the effectiveness of the “C” courses in preparing students 
for the WPR. 

It seems clear that there is a positive relationship between the number of “C” 
courses completed and success on the first WPR attempt. However, it is also clear that 
this relationship is not equally strong for all students. The effect is strong for ESL 
students, other non-native English speakers, those who enter through the DSP Program, 
and those who enter with lower Verbal SAT scores. With the exception of the sub-500 
VS AT group, completing the full sequence of “C” courses does not fully eliminate the 
difference in pass rates between these students and the overall group. However, we would 
expect that the gap would be closed considerably if the sequence had been completed. 
When a logistic regression is run for only those students who do not fit into any of these 
categories (N=466), there is no statistically significant relationship between the number 
of “C” courses completed and success on the WPR. 

Two groups of students completed significantly fewer “C” courses than their 
comparison groups. The DSP and the ESL students both completed “C” courses at 
significantly lower rates. It should be noted that these two groups are required to 
complete a number of courses that do not carry degree credit. It may be that they are less 
likely that other students to complete other “off track” courses even though the courses 
may be particularly beneficial for them. 

Given the widespread noncompliance with the old system, it is reasonable to 
believe that there will be similar behavior in the new system. If so, it is important that as 
resources are dedicated to tracking students and encouraging them to fully utilize the 
system, these resources and efforts should be focused on these groups of students who 
most benefited under the old system. 
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Abstract 

Recent conditions have nourished a competitive environment in which tuition and 
institutionally-funded discounts spiraled. Using data from 36 private liberal arts colleges, the 
authors investigated the interrelationship of market demand, institutional wealth, sticker price, 
and student profile to define an empirical model for estimating stress from increased unfunded 
tuition discounting. 



Introduction 



Rising Tuition 

The last two decades of the twentieth century were ones of increasingly intensive 
competition between postsecondary institutions. Economist Caroline Hoxby (in 
Trusteeship 2000) explained the increased competition as a combination of increased 
demand for high intensity education, decreased demand for low skilled labor workers, and 
increased student awareness and mobility. In another work she (Hoxby, 1997) also 
explained that the market for education became more competitive as it evolved into a 
nationally and regionally integrated market as students became more mobile. This 
competition has resulted in a drop in the market share of private institutions as students 
begin to choose lower-priced public alternatives to private education. At the beginning of 
the century, 4 out of every 5 students were enrolled in private colleges. The falling 
private sector market share has resulted in current private institution enrollment of 1 in 
every 5 students. 

Economic analyses have suggested that below-cost tuition at public colleges has 
drawn students away from the private sector (Wolfram 1997). McPherson (1978) found 
that the rise in the tuition gap between the public and private sectors accounted for nearly 
half of the enrollment shift. His regression analysis found the effect of public tuition 
levels on enrollment to be the strongest at less selective liberal arts colleges. This is 
extremely important for moderately selective colleges who are adversely affected by 
institutional density (public institutions per capita) and by low-tuition prices at public 



counterparts (Thompson and Zumeta 1998). McPherson and Shapiro (1994) pointed to 
liberal arts colleges in the Midwest as colleges which specifically faced enrollment 
declines and severe public sector competition. The same authors (McPherson and 
Shapiro, 1991) previously suggested that the elite and most-likely well-endowed private 
colleges provided sufficient financial aid to mute potential enrollment declines resulting 
from growing discrepancy between their prices and prices at potential public alternatives. 

Caroline Hoxby (1997) illustrated how the changing market structure described above 
explained tuition increases of 50% or more for selective, private colleges. The 
competitive environment drove institutions to raise their quality (described as vertical 
differentiation between institutions) in order to improve their market position. Clotfelter 
(1996) also demonstrated how elite colleges spend money to create “high quality,” 
indicating that an increase in quality cannot occur without subsequent increases in 
expenses and ultimately, tuition. As Russo and Coomes (2000) pointed out, private 
colleges are in a precarious position since they are tuition driven and thus must increase 
tuition to shoulder increased institutional costs. 

There also appears to be a relationship between institutional price and reputation of 
quality. Others (Duffy and Goldberg 1998) point to the “Chivas Regal Effect” which 
suggests that higher price equals higher superiority. This phenomenon is often cited as 
the explanation for the actions of many private colleges who significantly raised tuition to 
emulate the price increases of the Ivy League. Economist Charles Clotfelter (1996) also 
expressed that a major contributor to rising tuition costs was institutions’ “unbounded 
aspirations” to be “the best” and McPherson and Winston (1993) also acknowledge that 
high tuition itself may very well be a symbol of quality. 

An Environment for Increased Tuition Discounting 

Precedents in case law regarding the sharing of enrollment management information 
also nourished an environment in which sticker prices and institutionally funded 
discounts spiraled. When the Justice Department ended aid collaboration among the 
group of 23 highly selective colleges known as the Overlap Group in 1991, it intensified 
the competition among prestigious institutions. Although Congress did pass legislation 
which allowed institutions to agree not to engage in non-need merit competition, 
McPherson and Shapiro (1994) warned that the end of the overlap agreements would 
eventually cause merit scholarship competition to spread more widely among those elite 
institutions. 

Concurrently, a bull stock market and diversification of services in higher education 
led to unprecedented financial gains at many, although by no means all institutions. The 
strong market and the trend toward bolder and more diverse investments resulted in a 
move of assets from safe bonds into riskier, non-marketable securities pushed 
endowments up and resulted in several years of double digit returns on institutional 
investments. Historically, Breneman (2000) explained, annual endowment gains of 9 to 
1 1 percent, coupled with a 4 to 5 percent inflation rate and spending rates of 5 to 6 
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percent held endowments steady. However, recent year gains of over 20 percent, lower 
inflation (2 to 3 percent) and unchanged or even reduced spending have resulted in 
substantially increased endowment wealth at schools with the largest endowments. 

Increased competition and wealth led to radical changes in financial aid policy and 
practice at many wealthy institutions including shifts in grants from need to merit, 
replacement of loans with grants, and even tuition freezes. In order to stay marketable for 
quality students and to bolster their budgets, other institutions have followed suit, mostly 
by increasing the amount of institutional grant money offered to incoming freshmen. 
Colleges with a more precarious market position (low selectivity and low yield) began to 
increase their use of tuition discounting as an enrollment tool to improve their capacity to 
matriculate students. 

Prevalence of Tuition Discounting 

How prevalent is the practice of tuition discounting? The 1999 NACUBO Tuition 
Discounting Survey (as described in Lapovsky and Hubbell 2000) reported an average 
discount rate (defined by NACUBO as institutional financial aid dollars divided by gross 
tuition and fee revenue) of 37 percent among independent institutions. Tuition 
discounting has been on the rise; in the Fall of 1999, 79.4 percent of students received 
institutional financial aid, compared with 63.7 percent in the Fall of 1990. Additionally, 
the number of students who do not receive institutional financial aid dropped significantly 
between that same 10-year period. However, while the number of students receiving aid 
has increased, the average grant size as a percentage of tuition has remained relatively 
stable (increasing only 6.5 percent over the same 10-years). 

Institutional grants and discount rates vary widely in amounts across colleges. 
According to researcher Donald Basch (1996), these differences appear related to 
dissimilarities in selectivity, price, endowment, and location. Compared to the least 
competitive schools, the most competitive institutions (as characterized by Barron’s) were 
inclined to have lower percentage of needy students, lower average institutional grants, 
and ultimately, lower discount rates. He ranked colleges by price and noticed that 
although higher price was associated with a higher discount, the highest discount rates 
occurred at the 9 th and not the uppermost (10 th ) decile. He also noted that the percentage 
of needy students declined as price increased (not surprising since evidence, (Zemsky and 
Oedel, 1983) shows that family’s wealth is positively correlated with student’s academic 
qualifications at highly selective colleges); however the needy students at the higher- 
priced colleges tended to have more need. 

Basch (1996) also examined the relationship between endowment and discount, and 
suggested that a higher endowment allows an institution to provide more generous aid 
packages. His analyses revealed that higher endowment per student had a statistically 
significant positive effect on average institutional grant and discount rate. 



Problems with Tuition Discounting 

This growing trend of tuition discounting may prove stressful for less wealthy 
institutions, resulting in significant losses of tuition revenue which can be extremely 
detrimental for schools whose institutional grants are largely unfunded by endowment 
funds or annual gifts. The National Institute of Independent Colleges and Universities’ 
(N13CU) study revealed that endowment and gift revenue supplied only 30 percent of 
money necessary to support need-based grants in 1987-88, leaving 70 percent of grants 
funded by tuition and other revenue sources for most institutions (Evangelauf 1990). As 
the USA Group Foundation (NACUBO 2000) warns, high discount rates may leave 
institutions at risk of having fewer dollars accessible for educational expenses. 

The USA Group Foundation Study on tuition discounting (NACUBO 2000) found that 
institutions that offered the largest tuition discounts spent an average of $3,400 per 
student and lost at least $306 in per-student tuition revenue. A few highly selective 
institutions lost upwards of $800 per-student tuition dollars and a substantial loss was felt 
for at least 25 percent of institutions that discounted tuition. Losses in tuition revenue are 
dangerous as they may potentially lead to losses in academic spending. McPherson and . 
Shapiro (1998) demonstrated that between 1987 and 1994 financial aid increases wiped 
out a good portion of the private sector’s gross tuition increases. Forty-four private 
liberal arts colleges in their sample increased their sticker prices at an annual real rate of 
3.82% while tuition revenues increased at a rate of only 2.7% per year. 

Tuition revenue losses may be extremely dangerous to schools without large 
endowments to help fund discounts, especially if schools with smaller endowments have 
high discount rates; however schools with smaller endowment funds are just as actively 
providing institutional discounts. In fact, NACUBO’s most recent survey on tuition 
discounting (Lapovsky and Hubbell 2000) shows that no significant relationship exists 
between endowment size and level of tuition discount. Actually, there is a slight shift to 
higher levels of discount as endowment declines. The NACUBO study revealed that the 
difference in discount rates between the institutions with the largest endowments ($1 
billion or more) and those with smaller endowments (less than $50 million) is only 6.2 
percent, revealing that endowment size does not determine the size of the institutional 
discount. 

Estimating Stress 

This study investigated the interrelationship of market demand (selectivity and yield), 
institutional wealth (endowment per capita), sticker price, student need and percentage of 
in-state enrollment. We examined which of these factors predicted higher levels of 
unfunded institutional grants in an attempt to discern a measure for estimating 
institutional stress. By looking at predictors of unfunded institutional grants, we may 
determine potential stress indicators - levels that may alert an institution that it may be 
nearing a dangerous situation. 
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Methodology 



Data 

Data were collected from several sources. General institutional information was 
extracted from the Integrated Postsecondary Education Data System (IPEDS) Institutional 
Characteristics Survey of Fall, 2000 (National Center for Education Statistics, 2000). 

This included institution name, federal id code, state, and locale codes. Year 2000 
Carnegie classifications were merged with the remaining general information records 
from spreadsheets made publicly available by the Carnegie Foundation for the 
Advancement of Teaching (2000). 

Enrollment information was extracted from the IPEDS 2000 Fall Enrollment Survey 
(National Center for Education Statistics, 2000), including full- and part-time headcounts 
of degree-seeking students. These counts were used to estimate full-time equivalent 
(FTE) enrollment for each institution as full-time plus one-third of part-time enrollment. 
In-state enrollment counts were also derived from the IPEDS Fall Enrollment Survey by 
dividing the count of first-year students from in-state by the total first-year enrollment. 

Demand indices were estimated using data collected on the HEDS Consortium’s 
Freshman Admissions Survey for the fall of 2000. Rejection rates were calculated as the 
proportion of rejected students to total actionable applicants. Yield rates were estimated 
as the proportion of matriculating students to the count of admitted students. The demand 
index, which was used in our statistical models, was estimated as the log of the product of 
rejection and yield. 

Endowment data were obtained from the National Association of College and 
University Business Officers’ (NACUBO) Endowment Study (NACUBO, 2000). The 
numbers used were end-of-year market value of endowments as of July 1999. These 
values were divided by FTE for use in our statistical analyses, labeled wealth index, or 
endowment per capita. 

The unfunded institutional grant aid percentage was estimated from IPEDS Financial 
Survey (F2) data (National Center for Education Statistics, 2000). The proportion of 
unfunded grant aid was estimated as the proportion of total -unfunded student aid dollars 
to total institutional grant aid dollars. 

Aggregate student need, tuition and fees, and total institutional grants were extracted 
from the Freshman Financial Aid Survey collected by the HEDS Consortium in the fall of 
2000. Aggregate need and institutional grants were divided by FTE to estimate a per 
capita quantity. 

Three variables were transformed prior to analysis to make their distributions more 
normal. Sticker price was squared prior to analysis to correct for negative skewing 
whereas the square roots of in-state enrollment and endowment per FTE were used to 
minimize the effects of negative skewing in each of those variables. 
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Analyses 

We analyzed two models using multiple regression analysis: Model I, in which we 
predicted unfunded institutional grant value by the full predictor set including sticker 
price, need per FTE, in-state enrollment, endowment per FTE, and demand index. Model 
II was determined by removing terms from Model I via stepwise regression. For 
interpretive simplicity, given the disparate metrics of our predictors, we standardized all 
variables prior to analysis. 

Results 

Results for multiple regression analyses are summarized in Table 1. Results for 
Model I (Full Model) were not statistically significant ( Multiple R = 0.428, F = 1.301,/? = 
0.291) although the predictors explained more than 18% of the variance in unfunded 
institutional grant aid. 

Inspection of correlation coefficients (Table 2) revealed (a) a correlation between 
student need and unfunded institutional aid of 0.34; and (b) large correlations between In- 
state enrollment and Sticker price (r = -0.596), demand index and sticker price (r = 

0.573), endowment per FTE and in-state enrollment (r = -0.635), demand index and in- 
state enrollment (r = -0.520), and endowment per FTE and demand index (r = 0.300). 

Given the sizable correlations and the likely collinearity among the predictors, we ran a 
second model (Model D, Table 2) in which we eliminated predictors via backward 
stepwise progression. This produced a model (Model II in Table 1) which was 
statistically significant ( Multiple R = 0.342, F = 4.513, p = 0.041) in which need per FTE 
predicted 12% of the variance in unfunded institutional aid [/ 3 = 0.314(0.148), t = 2.124,/? 
= •041], 

Figure 1 reveals the relationship of need per FTE and unfunded institutional grant aid. 
Although the pattern was linear, the regression line did not cleanly fit the observed 
pattern of the scatterplot. Endowment per FTE, represented by the size of the pips 
seemed to be unrelated to this relationship. 

Because of the expectation of a relationship between sticker price and institutional 
aid, a follow-up plot of these variables was constructed to determine if the relationship 
was nonlinear, thus obscuring the relationship (Figure 2). In this plot, a contour was 
overlaid revealing clustering in the data. Interestingly, there appeared to be two distinct 
clusters of institutions with parallel linear patterns, suggesting some differentiation of aid 
strategy based on some other criteria. In this plot, pip sizing based on need per FTE did 
not suggest that it adequately explained the relationship. 

Figure 3 represents the same data as in Figure 2 replotted with pip sizes determined 
by the demand index. This plot clearly suggests that demand may be a differentiating 
factor among the two groups of institutions. However, some institutions with similar 
demand and pricing still appeared to cluster differently based on some unobserved 
characteristic. 

In Figure 4, we replotted the same information with the percentage of students from 





in-state as the pip sizing criteria. This sizing parameter suggested another potential 
predictor of the apparent parallel linear clusters in the data. A comparison of Figures 3 
and 4, suggests that the in-state enrollment variable, along with the demand index may 
adequately explain these apparent clusters. 

To test these observations, we ran a final multiple regression analysis in which we 
used a dummy variable to represent the two clusters of institutions suggested by our 
model in addition to all previously included predictors (Table 3). This model fit the data 
much better than the first models, explaining more than 70% of the variance in unfunded 
institutional grant aid [Multiple R = 0.840, F = 14.841, p < .001). Interestingly, stepwise 
fitting dropped the only predictor that was included in Model II - need per FTE. It is 
noteworthy that in-state enrollment and demand Index, while included in the final model 
specification, were not statistically significant, possibly confirming their importance in 
creating the two clusters that we observed. 



Discussion 

Although student need predicts unfunded institutional grant aid as expected, it does 
not adequately explain the variance in these aid amounts and, by extension, the policies 
that underlie them. This suggests that as institutions move from largely need-based grant 
aid to more merit aid as an enrollment tool, factors other than student need will explain 
the varying amount of grant aid supplied to incoming freshmen. 

Viewed simply, the relationship between sticker price and unfunded institutional aid 
appears to be non-linear. However, when inspected closely, it appears that there are 
parallel linear patterns for two distinct groups of institutions: one group that has high 
demand and gives less aid per student and a second group that has less demand and gives 
larger grants per student. This supports Basch’s (1996) finding that although higher price 
was associated with a higher discount, the highest discount rates occurred at the 9 th and 
not the 10 th decile of colleges ranked by price. A more complex situation is setting the 
amount of unfunded institutional grant if need and price alone do not predict the amount. 

Aside from demand, the percentage of students from within state seems to best 
differentiate the two groups that were observed in the plots of our study. This seems to 
confirm earlier research that revealed that low tuition at public institutions adversely 
effected enrollment and tuition prices at private colleges who directly competed with 
those in-state public schools for enrollment. Private colleges that draw a large base of 
their students from in-state populations must increase grant amounts to reduce 
competition from in-state public universities. 

Contrary to expectations, there seemed to be little relationship between institutional 
wealth and grant size. However, given the limited focus of our study on unfunded 
institutional grant aid as a potential indicator of stress, we necessarily ignored another 
potential stressor that is likely more closely related to wealth - tuition dependence. That 
is, an institution with low demand, drawing heavily from within state, giving larger 
unfunded grants, and heavily dependent on tuition revenues is potentially on shakier 
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fiscal footing than a similar institution that has more endowment returns with which to 
pay the bills. 

When included as a predictor, the clustering that was observed in the plots improved 
the overall fit of the predictive model and explained 55% more of the variance than the 
best model that did not include some indication of the clustering. Although we did not 
rigorously investigate the factors that determine institutional clustering, our observations 
of regression results and of the graphics suggests that private colleges that experience 
moderate to low demand and recruit heavily from within state will likely recycle tuition at 
greater rates as the sticker price is increased. This suggests that these factors should 
perhaps be monitored closely, especially for moderately selective institutions 

Future directions 

This study was a limited “first look” that was focused on unfunded institutional grant 
aid as a stress criterion. As was noted above, there are potentially many other factors to 
investigate as stressors. Most especially, tuition dependence is an important factor that 
warrants further investigation. Many of the trends in tuition recycling that were examined - 
here become more critical at institutions where tuition makes up a significantly higher 
proportion of total revenues. 

In future studies, we hope to examine the clusters we observed here more closely. 

Are the clusters real or merely an artifact of the limited variable space defined by the 
current study? What are the determinants of membership in one cluster or the other? Are 
there questions of institutional policy - manipulate factors - that can or do determine 
migration to one pattern of behavior or the other? If all factors are exogenous, what are 
the key indicators to watch closely in determining one’s institution’s “footing?” 

It is also important to examine the patterns of non-need aid more closely. In this 
study, we largely ignored this question in an effort to keep our initial model simple - 
especially in light of our small sample size and limited degrees of freedom. However, the 
question of a shift from need-based to non-need-based aid becomes more critical to a 
correct interpretation of the results observed in this study. The elimination of student 
need from the final regression equation suggests that factors other than federal or 
institutional need methodology are playing an increased role in disbursement of student 
aid at less selective institutions, especially as sticker price rises. 

Two factors particularly limited this study. First, it was limited to a snapshot of a 
single year’s indicators and second, the dataset was populated by largely moderately to 
highly selective colleges. A multi-year analysis will permit us to look at direction of 
change as another dimension of stress. In future studies we intend to expand our dataset 
to include more moderately-low and lowly selective colleges to provide more variance in 
the predictors and outcomes. 
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Table 1 

Results of Regression Models I and II 



Model I: Full Model 



R: 0.428 

R 2 : 0.183 



coeff. se 



P 



Intercept 0.027 

Sticker Price 0.179 

Need per FTE 0.252 

In-state enrollment 0.336 

Endowment per FTE 0.228 

Demand Index 0.166 

ANOVA 

Source 

Regression 5.387 

Residual 24.007 



0.156 


0.176 


0.862 


0.241 


0.746 


0.462 


0.183 


1.381 


0.178 


0.298 


1.127 


0.269 


0.237 


0.960 


0.345 


0.224 


0.742 


0.464 



df 


ms 


F 


5 


1.077 


1.301 


29 


0.828 





Model II: Model Resulting from Stepwise Regression 

R: 0.342 

R 2 : 0.117 



coeff. 



Intercept -0.004 

Need per FTE 0.314 

ANOVA 

Source ss 

Regression 3.460 

Residual 26.064 



se t p 

0.146 -0.026 0.979 

0.148 2.124 0.041 



df ms F 

1 3.460 4.513 

34 0.767 




t n 



6 



P 

0.291 



P 

0.041 
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Table 2 

Correlation coefficient matrix and criterion and predictor variables 





(1) 


(2) 


(3) 


(4) 


(5) 


Avg. Unfunded Grant (1) 


1.000 










Sticker Price (2) 


0.132 


1.000 








Need per FTE (3) 


0.337 


0.035 


1.000 






In-state enrollment (4) 


0.067 


-0.596 


0.236 


1.000 




Endowment per FTE (5) 


0.110 


0.193 


0.044 


-0.635 


1.000 


Demand Index (6) 


0.105 


0.573 


-0.210 


-0.520 


0.300 



Table 3 

Model HI: Adjusted Model Including Clustering 

R: 0.840 

R 2 : 0.705 





coeff. 


se 


t 


P 


Intercept 


1.315 


0.202 


6.514 


0.000 


Cluster Dummy 


-2.178 


0.294 


-7.402 


0.000 


Sticker Price 


0.947 


0.145 


6.534 


0.000 


In-state enrollment 


0.269 


0.176 


1.533 


0.136 


Endowment per FTE 


0.297 


0.142 


2.089 


0.045 


Demand Index 


0.163 


0.132 


1.243 


0.223 


ANOVA 


Source 


SS df 


MS 


F 




Regression 


26.004 


5 


5.201 


14.841 


Residual 


10.863 


31 


0.350 





Figure 1. 

Relationship of Need to Unfunded Institutional Grant Size. 
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Figure 2. 

Relationship of Sticker Price to Unfunded Institutional Grant Size. 
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Figure 3. 

Relationship of Sticker Price to Unfunded Institutional Grant Size. 




Sticker Price 





- 108 - 



Average Institutional Grant 



Figure 4. 

Relationship of Sticker Price to Unfunded Institutional Grant Size. 
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THE IMPACT OF INSTRUCTIONAL DELIVERY ON 
LEARNING OUTCOMES AND INTENT TO PERSIST 



Kristin A. Owens 
University of Maryland 

J. Fredericks Volkwein 
Pennsylvania State University 



Introduction and Problem Statement 

The academic disciplines of criminal justice and higher education have provided 
little evidence to document the effects of educational participation during inmate 
incarceration (Almond, 1989; Boaz, 1976; Cheatwood, 1988; Wolf & Sylves, 1981). 
Research has been hindered by changing societal beliefs about the rehabilitative potential 
of incarceration, as well as by political and program funding decisions. Pell Grant 
eligibility and funding for inmates was eliminated in 1994 with the passage of the Violent 
Crime Control Act. The lack of valid, rigorous research contributed to the downfall of 
programs since there was little hard evidence in favor of maintaining their existence. 

Nevertheless, the enormous growth in the American prison population and the 
growing importance of educational opportunity and attainment for both individuals and 
society gives some weight to the need for such research. The theoretical purposes of both 
traditional higher education and corrections are complimentary; higher education attempts 
to provide an environment for gaining new knowledge, skills and educational 
advancement, while corrections applies the principal that attitudes, ideas and behavior 
can be changed through rehabilitation (Gehring, et al, 1998). However, countless 
research studies of effectiveness in traditional higher education continue to be produced 
while college level programs in correctional facilities lack proper evaluation. Few have 
attempted to apply the current student outcomes models to inmate learners, and decisions 
about funding for post secondary programs in prisons are often made in the absence of 
alternatives supported by evidence. 



Conceptual Frameworks 

In designing this study, we drew upon appropriate frameworks from the scholarly 
literature. The most traditional view is that pre-college characteristics like student family 
backgrounds, academic preparedness for college, and clear goals are the main factors 
accounting for differences in academic performance, persistence behavior, and other 



educational outcomes (Feldman & Newcomb, 1969; Astin, 1991; Stark et al. 1989; 
Willingham et al. 1985). 

A second group of alternative yet complementary perspectives fall under the 
genera] description of student-institution fit models (Pascarella & Terenzini, 1991). 
These models generally suggest that student persistence and growth depends on the 
degree of successful integration into the academic and social structures of the institution, 
and on the amount of student involvement and effort. 

A third relevant framework for this study derives from the literature on self- 
efficacy — a person’s judgment of their capabilities to act in order to attain their desired 
goal or performance (Bandura 1986). Self-efficacy affects choice of activities, goal 
formulation, effort and persistence to degree attainment (Bandura, 1977; Lent, Brown & 
Larkin, 1984; Schunk, 1991). Research on inmates has found that the longer a student is 
in prison, the lower the self-efficacy scores (Parker, 1990). 

Based upon these three branches of the scholarly literature, we assume that inmate 
educational outcomes and reported gains and intent to persist are Bi-products of the 
following factors: demographic backgrounds (including age, sex, and ethnicity), length of 
prison sentence, instructional method (traditional classroom versus distance), and 
learning context (faculty effectiveness in the classroom, peer interaction, and learning 
environment). This research focuses on the role of instructional method, controlling for 
these other potential influences. 

Research Design and Methodology 

This study utilizes a cross-sectional research design, collecting information at a 
single point in time. The subjects of this study are 274 inmates (out of 279 enrolled) from 
nine correctional institutions in a single state. The survey was administered to 111 
students in traditional courses and 163 students in distance education courses. The survey 
was kept confidential and completely voluntary. The inmates completed the 
questionnaire within twenty minutes. Several factors prevented the study from following 
a longitudinal design, including restrictions on information access, confidentiality, and 
the extreme transience of the prison population due to probation and parole, prison 
transfers, and programming re-assignments 

Based on the outcomes and self-efficacy literature, 44 survey items were designed 
in the areas listed in Figure 1. In each category, item construction was grounded in theory 
and research to ensure construct validity. To ensure face or content validity, we consulted 
experts in higher education and criminal justice, and used a practitioner focus group to 
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assist with item formulation. Survey items were developed, pilot tested, and subsequently 
revised. 

Demographic Information includes age, race, gender, length of sentence, and work hours. 

Instructional Method- Students enrolled in distance education courses were coded ‘1’, 
traditional classroom courses coded ‘O’. 

Faculty Effectiveness- This six-item scale measures the nature of classroom interaction 
between inmates and their instructors. These six items are drawn from the classroom 
involvement scale developed by Terenzini, et al (1982, 1984) and adapted for this study 
(alpha= .89). 

Peer Interaction- This five-item scale measures the nature of the interactions between 
inmates and their student peers. These five items are drawn from the peers sub-scale of 
the Mattering Scales For Adult Students in Post-secondary Education (Schlossberg et al., 
1990) and modified to meet the needs of this study (alpha=.83). 

Learning Environment- This scale is measured by five items describing the learning 
environment, drawn from the Learning Environment Inventory (Fraser, Anderson, 
Walberg, 1982), originally designed to assess the classroom learning climate of secondary 
students, and later adapted to successfully assess the learning environment of college 
classrooms (Ellet, 1976; Kent & Fisher 1997). The five items were adapted to meet the 
needs of this study (alpha=.69). 

Educational Outcomes- As suggested by Figure 1, a scale of educational outcomes is 
treated first as a dependent variable, then as an independent predictor of persistence. The 
outcomes scale is measured by six domains describing career preparation, gains in job 
skills, problem solving, openness to new ideas, control, and civil responsibility. These 
six items are drawn from the Noel-Levitz College Student Inventory (Stratil & Schreiner, 
1993) and modified to meet the needs of this study (alpha=.86). 

Intent to Persist - Two items based on Bandura’s concept of self-efficacy, were used to 
measure intent to persist in the course. Students rated their confidence in completing the 
course and semester on a 1 to 5 scale (alpha=.76). Three items also based on Bandura’s 
concept of self-efficacy, were used to measure intent to persist towards a degree. 

Students rated their confidence in persisting to degree attainment on a 1 to 5 scale 
(alpha=.92). 





Figure 1; Research Desien Model 
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Results for Educational Outcomes 

We analyzed the relationships among the variables using OLS regression and 
entered the variables in blocks with listwise deletion of cases. The results are shown in 
Table 1 and summarized below. The final equation (Model 4) is robust and accounts for 
almost 36% of the variance in perceived outcomes. The findings from this highly 
controlled population of prison inmates look very similar to the results we typically have 
found in outcomes studies of traditional college students. 

• Being female is positively associated with Educational Outcomes. 

• Faculty classroom effectiveness is highly influential and positive. 

• Peer Interaction is highly influential and positive. 

• Inmates receiving distance instruction report significantly lower 
outcomes, but the influence is indirect. This particular type of distance 
instruction exerts a direct negative influence on the measures of the Learning 
Context, especially Faculty Effectiveness and Peer Interaction, and they in turn 
influence Outcomes. 

• Ethnicity, age, length of prison sentence, and hours of work and study are not 
significant. 
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Table 1: OLS Regression Results for Educational Outcomes 



Variable Block 


Model 1 


Model 2 


Model 3 


Model 4 




Std. Beta 


Std. Beta 


Std. Beta 


Std. Beta 


1. Demographics: 










Female 


.229** 


.228** 


.136* 


.113* 


Minority 


-.022 


-.022 


.003 


-.041 


Age 


-.026 


-.025 


.050 


.007 


Work Hours 


-.077 


-.076 


-.036 


-.072 


2. Length of Sentence 




-.006 


-.006 


-.067 


3. Distance Education 






-.252** 


-.066 


4. Learning Context: 










Faculty Effectiveness 








.262** 


Peer Interaction 








.379** 


Learning Environment 








.066 


Study Hours 








.002 


Total R2 


.052** 


.052** 


.101** 


.357** 



* = p<.05 ** = p<.01 



Results for Intent to Persist in Course 

We analyzed the relationships among the variables using OLS regression and entered 
the variables in blocks with listwise deletion of cases. The results are shown in Table 2 and 
summarized below. The final equation (Model 5) is robust and accounts for 37% of the 
variance in intent to persist in the course/semester. 

• Gender, Ethnicity, Age, Sentence Length, and hours of work and study are 
not significant 

• Distance education students not only report lower Educational Outcomes 
also report significantly lower Intent to Persist in Courses. This particular 
type of distance instruction interacts negatively with the measures of Learning 
Context, especially Faculty Effectiveness, but also Peer Interaction. 

• Faculty Effectiveness is highly influential and positive —twice as 
influential as learning outcomes and three times as influential as mode of 

instruction. 

• As we saw in Table 1, Peer Interaction exerts a direct positive influence 
on the outcomes measure, but in Table 2 peer influence on Intent to Persist in 
Course is indirect, as its influence disappears once the Outcomes measure is 
added in Model 5. 

• There is a strong positive statistical connection between perceived 
Outcomes and Intent to Persist in the Course/semester. 
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Table 2: OLS Reeression Results for Intent to Persist in Course/Semester 



Variable Block 


Model 1 


Model 2 


Model 3 


Model 4 


Model 5 




Std. Beta 


Std. Beta 


Std. Beta 


Std. Beta 


Std. Beta 


1. Demographics: 












Female 


.103 


.103 


.080 


.080 


.055 


Minority 


-.042 


-.042 


-.035 


-.031 


-.022 


Age 


-.014 


-.014 


.005 


-.084 


-.086 


Work Hours 


.058 


.058 


.067 


.012 


.028 


2. Length of Sentence 




.000 


.000 


-.044 


-.029 


3. Distance Education 






-.064 


.154** 


.168** 


4. Learning Context: 












Faculty Effectiveness 








.528** 


.470** 


Peer Interaction 








.183** 


.100 


Learning Environment 








-.009 


-.023 


Study Hours 








.082 


.081 


5. Outcomes 










.219** 


Total R2 


.017 


.017 


.021 


.339** 


.370** 



* = p<.05 ** = p<.01 



Results for Intent to Persist in Degree 

We analyzed the relationships among the variables using OLS regression and entered 
the variables in blocks with listwise deletion of cases. The results are shown in Table 3 and 
summarized below. The final equation for degree persistence (Model 5) is not as robust as 
for outcomes and course persistence, and accounts for 15% of the variance. 

• Age is significant. Results show that younger students (30 years and younger) 
are more likely to indicate positive Intent to Persist in Degree than older adults 
(older than 30 years) 

• Being female directly influences Intent to Persist in Degree through Model 3. 
Once Learning Context is added to the model, Gender is no longer significant. 

• Ethnicity, Work Hours and Sentence Length are not significant. 

• Distance education has no effect on Intent to Persist in Degree. 

• Results indicate inmates who experience gains in Educational Outcomes are 
also likely to report higher Intent to Persist towards a Degree, controlling for 
other factors. 



Table 3: OLS Regression Results for Intent to Persist in Pe 2 ree 



Variable Block 


Model 1 


Model 2 


Model 3 


Model 4 


Model 5 




Std. Beta 


Std. Beta 


Std. Beta 


Std. Beta 


Std. Beta 


1. Demographics: 












Female 


.156* 


.154* 


.144* 


.130 


.098 


Minority 


.104 


.104 


.107 


.077 


.089 


Age 


-.146* 


-.144* 


-.136* 


-.155* 


-.157* 


Work Hours 


-.012 


-.010 


-.006 


-.024 


-.004 


2. Length of Sentence 




-.009 


-.009 


-.012 


-.007 


3. Distance Education 






-.026 


.026 


.045 


4. Learning Context: 












Faculty Effectiveness 








.013 


-.059 


Peer Interaction 








.202** 


.097 


Learning Environment 








.021 


.003 


Study Hours 








.104 


.104 


5. Outcomes 










.277** 


Total R2 


.047* 


.047* 


.048* 


.104** 


.154** 



* = p<.05 ** = p<.01 



Conclusions 

The phrase “culture of failures” (Roundtree et al, 1982: 17) is often used when 
describing the inmate population; however, “if an educational approach can help modify 
this sense of failure, such an approach deserves attention.” Since self is an important 
determinant of personal adjustment among minority college students, especially inmates, 
sources of positive self-expectations can be modeled in higher education programming 
within correctional institutions (Solberg & Villarreal, 1997). The findings of this study 
should encourage change in existing correctional education program structures, by 
providing more opportunities for faculty and peer interactions. 

There is a belief in the correctional community that educational programs are 
wasted on long-term prisoners and that they are not motivated to learn. This study finds 
that sentence length is not significantly associated with student outcomes or intent to 
persist. As shown in the statistical analysis, inmates with varying sentence lengths 
reported similar gains in outcomes and intent to persist in both course and degree. 
Admission criteria for inmate access to college programs should reflect these findings. 
Program participation should not be limited by sentence length, as it is now in most 
states. Inmates incarcerated for longer periods of time should not be denied post- 
secondary educational opportunities just because they will not be released anytime soon. 

Another important finding in the study relates to gender. As a whole, there are far 
fewer female correctional facilities across the nation. Appropriate staff, learning 



environments, and materials may be disproportionate when compared to male facilities. 
Since this study indicates that females are associated with positive outcomes and intent to 
persist, college programs should be encouraged to expand for women audiences within 
the correctional system. All incarcerated individuals should be afforded the same level of 
education experiences, regardless of gender. 

Our research suggests that traditional classroom instruction is superior to video 
instruction for student inmates. For reasons of both budget constraints and security, 
correctional distance education contributes to rely substantially on correspondence 
courses and videotapes. Our findings indicate that videotape delivery, as distinct from 
face-to-face instruction, results in lower educational outcomes and lower intent to persist. 
This finding is consistent with what we know about the value of active versus passive 
learning. While we are skeptical about the effectiveness of distance education for this 
population, we are not necessarily prepared from this study to draw conclusions about 
distance education using more advanced technologies and synchronous learning. Method 
of instruction has never before been examined as an indicator of inmate educational 
outcomes, nor as a contributor to low recidivism. Thus, more research is needed on 
prison culture, instructional method, and inmate learning. 

An important conclusion determined by the literature review and this study is that 
post-secondary educational outcomes, result from multiple influences (Wolf & Sylves, 
1981). Studies on traditional college campuses and prisons alike have shown that 
multiple variables contribute to student success, learning, and persistence. Further 
research is needed with inmate students to draw the connection between education and 
reduced recidivism. 

In summary, this study examines the relationships among instructional delivery 
method, student outcomes, and intent to persist in both course and degree for a population 
of 274 inmates participating in college programs at nine Maryland State prisons. 
Consistent with the existing higher education research and literature, faculty classroom 
effectiveness and inmate peer interactions exert the strongest positive and most direct 
influences on educational outcomes; and these outcomes in turn have the strongest impact 
on persistence. Video delivery, as distinct from traditional face-to-face instruction, has a 
direct negative effect on faculty and peer interactions, and an indirect negative effect on 
educational outcomes, and on intent to persist. Being female is positively associated with 
higher outcomes and intent to persist, encouraging the continued and expanded college 
programming opportunities for women inmates. An unexpected finding is the 
insignificance of sentence length on both student outcomes and intent to persist, thus 
supporting more flexible admission criteria for post-secondary correctional programs. 
Finally, the measures of ethnicity, work hours, and study hours are not influential 
predictors. 

With the continued growth of technology and new distance education avenues, the 
opportunities for new and enhanced programs have increased. Educators and program 
administrators need not answer how can we teach utilizing distance instruction 
technology, but instead, how can we maximize student learning (Champagne, 1998: 90). 
Hopefully this study will enable more positive changes and informed decision-making 
opportunities for practitioners in the field of correctional education and encourage 
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additional areas of research for academics engaged in student development theory. 
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THE IMPACT OF LOTTERY INCENTIVES ON STUDENT SURVEY 

RESPONSE RATES 
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Abstract 

Lottery incentives are widely used by institutional researchers despite a lack of 
research documenting their effectiveness. A controlled experiment tested the effects of 
lottery incentives using a prospective college applicant web survey, with emails sent to 
over 9,000 high school students. The impact of the level of lottery incentive on response 
rates, time to response, and response bias is discussed. 



Introduction 

Student survey data have grown increasingly important for institutions of higher 
education. But as the use of student and alumni data have increased, response rates to 
surveys have been falling nationally (Dey, 1997; Smith, 1995; Steeh, 1981). Survey 
fatigue is commonly cited, as public opinion polls have become more popular with the 
media and telemarketers use surveys for data mining research. Increasingly educational 
researchers are faced with the prospect of simply maintaining, rather than increasing, 
survey response rates. 

As response rates continue to shrink, researchers face increasing costs to counter 
survey non-response. Second and third mailings, for example, must be larger if the initial 
mailing elicits a weak response pool. Given that survey research is one of the most common 
activities in institutional research (Schlitz, 1988), researchers must refine their data collection 
tools to counter this growing trend. 

Based on a survey of colleagues, lottery incentives appear to be a popular method for 
increasing response rates in institutional research surveys. A lottery incentive is a reward 
offered to survey recipients for responding to a survey, in which every recipient who 
responds is entered into a drawing (similar to a lottery) for one or more prizes. 

In addition, the growing use of electronic surveys may be leading to an increased use 
of lottery incentives in survey research. Unlike mailed surveys, it is impossible to include 
incentives such as a dollar bill with an email survey or email notice about a web survey 
(Couper, 2000). Incentives paid upon completion are possible with electronic surveys, 
however, as these can always be mailed to respondents. With the growing use of web 
surveys the need to understand the efficacy of lottery incentives has increased. 

While lottery incentives appear to be a popular and perhaps growing method for 
increasing response rates, the research literature on lottery incentives indicates they have 
little or no impact on survey response. When applied to higher education, however, this 



research may not be relevant. Previous studies have been conducted on members of the 
general population, and it may be possible that college students are more price-sensitive 
than the average person. If so, lottery incentives may have an impact on response rates in 
student surveys while not having any impact on surveys of the general population. 

Educational researchers have investigated the impact of questionnaire format 
(Boser, 1990), survey length (Adams & Gale, 1982), bad addresses (Grosset, 1995) and 
multiple follow-ups (Cote et al., 1986; Smith & Bers, 1987) on student and former 
student survey response, but no research has been conducted studying the impact of 
lottery incentives on response rates. The lack of research in this area is not surprising. 

To truly test the effectiveness of incentives, at least two randomly selected groups of 
students must be used: a control group receiving just a survey, and one or more 
experimental groups receiving both a survey and an incentive. Comparing response rates 
across different administrations of the survey (e.g., across years) does not work, as other 
factors may affect response rates besides a change in survey administration. A similar 
rationale holds for surveys across institutions within a university system. 

Yet such an experimental situation poses a problem for any college or university, 
as students in the control group will discover that other students have the possibility of 
winning a prize for filling out the same survey. Given communication among students 
and the likely negative reaction of the control group, successful implementation of a 
controlled experiment of incentives is simply not possible on our campuses. 

We circumvent this problem by conducting a controlled experiment on high school 
students who have contacted an admissions office at a small, liberal arts college for 
information about applying. Since this college draws students from across the nation, we can 
split the survey into groups without worrying about communication between groups. And 
because the high school students are seniors about to attend college, their sensitivity to a 
lottery incentive should be much more similar to the typical college student’s sensitivity 
compared with a member of the general population. 

Our research is useful given the prevalent use of lottery incentives in institutional 
research, the lack of evidence of their efficacy, and the probable increase in their use as 
researchers begin moving from paper to electronic surveys. The paper consists of four 
parts. After first reviewing the literature on lottery incentives and response rates, we 
determine the prevalence of such incentives in institutional research. We next describe 
the experiment and how it was conducted. We then analyze the data for differences in 
response rates and time to response. We also examine item non-response differences 
between the experimental groups. 



Literature review 

Incentives are theorized to affect response rates by affecting how the respondent 
views the costs and benefits of the survey process. Social exchange theory proponents argue 
that by providing general, diffuse, or “token” benefits to potential respondents, researchers 
can convince potential respondents “that in the long run the anticipated benefits of 
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responding outweigh the anticipated costs,” (Dillman et al, 1996, p. 378). Such benefits can 
include small amounts of money (usually in the $l-$5 range), mention of benefits that will 
accrue to groups to whom the respondent belongs, assistance to the research sponsor, or 
benefits to the population as a whole (Dillman et al., 1996). 

Economic exchange theory, on the other hand, posits that respondents will fill out and 
return surveys in exchange for specific monetary compensation, rather than a broader, more 
diffuse benefit. Dillman (2000, pp.14-15) argues that economic exchange simply does not 
work with surveys. As evidence he cites survey research that promised payment to 
respondents upon completion, and which found no increase in response rates using this 
method. 

If Dillman were correct, then we would expect lottery incentives to have no effect on 
response rates. The issue here is that the payment of the incentive after the survey has been 
returned turns the survey process into an economic transaction rather than a social one. 
Prepaid incentives work precisely because they are obtained without any effort on the part of 
the respondent, and because they may create a sense of duty on the part of the respondent. 
This norm of reciprocity (Groves et al., 1996) arises because the token incentives are viewed 
as a gift rather than compensation for effort. 

Alternatively, an economist might argue that lottery incentives do not work simply 
because the benefits appear too diffuse. With a lottery the expected benefit is not the 
monetary amount of the incentive, but the amount multiplied by the probability that the 
respondent will be selected a winner in the lottery. The implication is that larger lottery 
incentives might have an impact, as they will have a larger expected value for the respondent. 
(Alternatively, increasing the probability of winning should also have the same effect.) An 
additional complication here is to what extent the respondent actually believes that a lottery 
exists and will be run fairly. 

The empirical research on incentives indicates a conclusive positive impact on 
response rates; however, this impact very much depends on the type of incentive. Incentives 
can be divided into two groups based on when the survey recipient receives the incentive: 
either with the survey (known as pre-payment) or after the survey has been completed and 
returned (post-payment). 

Numerous studies have been conducted studying the impact of pre-paid incentives on 
survey response, and the results indicate that their use almost invariably increases response 
rates (e.g., Church, 1993; Singer et al., 1999; Willimack et al., 1995; Zusman and Duby, 
1987). Less certain is the impact of post-paid incentives. Several experimental studies have 
been conducted that compare the impact of pre- and post-payment of incentives, with the 
general finding that post-payments have no statistically significant impact on response rates 
(Church, 1993; James & Bolstein, 1992; Singer at al., 2000). Other researchers have tested 
the effect of lottery post-payments, in which the incentive is not guaranteed but is instead 
dependent upon the outcome of a drawing. These researchers have found no effect for lottery 
incentives (Warriner et al., 1996). 



The use of lottery incentives in institutional research 

Given a lack of theoretical and empirical support for the use of lottery 
incentives in surveys, the extent of their use in institutional research is surprising. In Spring 
2000 we conducted a short web survey asking institutional researchers about their use of 
lottery incentives. Members of seven regional institutional research listservs (California, 
Mid-America, Northeast, Pacific Northwest, Rocky Mountain, Southern, Upper Midwest) 
were notified of the survey, with 374 people responding. Respondents were asked several 
questions about the surveys they conduct, and whether their institution was public or private. 

Table 1 shows the number of surveys in which researchers used a lottery 
incentive during a typical academic year. Overall about a third of the respondents administer 
at least one survey a year that uses a lottery incentive, with about half of the respondents at 
private institutions using lottery incentives at least once a year. 

Researchers were also asked what types of incentives they use. Table 2 lists 
the type of prize. Monetary prizes (either cash or gift certificates) tend to predominate, with 
respondents at private institutions more likely to list these as the incentives they use. The 
differences between private and public institutions seen in Tables 1 and 2 most likely stem 
from differential resources, with private institutions using monetary incentives and using 
them more often than their public counterparts. 

Researchers were also asked their opinion of the effect of lottery incentives on 
response rates. Table lists the responses for all respondents, and only those reporting that 
they used a lottery incentive in at least one survey per year. Given the prevalent usage seen 
in Table 1 , not surprisingly 75% of respondents reported that they believed lottery incentives 
increase response rates, with the vast majority indicating that they ‘somewhat’ rather than 
‘greatly’ increase response rates. About 90% of respondents who indicated they use lottery 
incentives in their surveys believe incentives positively increase response rates, while over 
half of those who do not use incentives still believe in their efficacy. 

In sum, although the literature on incentives and response rates shows that 
post-payment of incentives in general and lotteries in particular have little or no impact on 
survey response, use of such lottery incentives appears common in institutional research. 
The remainder of the paper investigates whether lottery incentives are indeed effective when 
used with student surveys. 

Research design 

The experiment was conducted in Spring 2001 during a survey of non-applicant high 
school students. These prospective students had contacted the institution for information 
about the institution during the previous year, but did not apply for admission. Of about 
13,000 prospects, 9,305 had provided enough information about their high school during the 
contact to allow the assignation of the appropriate CEEB code for their high school. Because 
it was essential that members of the control group did not discover that other students had 
been offered an incentive for response, students were grouped by high school for the 
experiment. The average number of students per high school was 2.64, with the number of 
students ranging from 1 to 93. 
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The high school codes were randomly divided into five groups: a control group and 
four incentive groups. The high schools codes were then used to assign students to an 
experimental group. This ensured that students in the same high school were placed into the 
same experimental group, and therefore would not discover via communication with friends 
that others in their high school had received a different incentive offer. Table 4 shows the 
number of high schools and number of students in each experimental group. 

The survey administration consisted of an initial email notification with an embedded 
survey link, and each group was administered the same survey. Although students were 
asked to enter their email address during the survey, the sample groups were given links to 
five separate websites to ensure we could track differences between groups. The four 
randomly selected incentive groups were informed that if they responded to the survey they 
would be entered into a drawing for a $50, $100, $150 or $200 gift certificate to 
Amazon.com, depending on the group. The emails were identical except for the incentive 
group emails, which included this passage about the lottery incentive: 

Because we realize your time is valuable, when you complete the survey you will be 

entered into a drawing for a $ gift certificate from Amazon.com. The drawing will be held 

within six weeks and you will be notified of the outcome via email. 

The initial email was followed three days later with a reminder email to non- 
respondents, with a final reminder to non-respondents five days after the first reminder. Each 
reminder included details about the incentive for each group. 

Results 

In all analyses we examined the control and experimental groups to test three 
main questions: 

• Do all five groups (the control and four levels of incentive) differ from one 
another? 

• Does each incentive level individually differ from the control group? 

• Do respondents offered an incentive differ from those not offered and 
incentive? 

The first question tests if increasing levels of incentives have a differential impact, 
in other words, do response rates increase as the amount of the incentive increases? This 
is the most common view of incentives and their impact on response rates: more is better. 

The second question tests if only some of the incentives have an impact. For 
example, there may be a nonlinear relationship between response rates and incentives 
amounts. Small amounts may have little impact because the respondent does not feel 
they are adequate to justify his or her expenditure of time. Large amounts, on the other 
hand, may have little impact because respondents are skeptical they will receive the prize 
given the large value. Alternatively, large amounts may be viewed as compensation 
rather than a token benefit, thus transforming the relationship between one of reciprocity 
to an economic one. • 



The third question simply tests the overall impact of offering an incentive. There 
may not be much of a difference in response rates between the $50 group and the $200 
group, and depending on the data, ANOVA testing for differences between all five groups 
could result in a null finding. Yet an ANOVA testing the control versus all the incentive 
groups might detect a positive impact, so this third hypothesis is simply another way to 
check the data. 

Table 5 shows the response rates for the initial email and the response rates at the 
close of the experiment. Rates are shown for all five groups, for the four incentive groups 
combined, and for the entire sample. Overall 15.2% of the sample responded to the 
survey. Differences between the control group and incentive groups were quite small. 
Almost 14 percent of the control group responded, while overall 15.6% of respondents in 
the incentive groups participated in the survey. 

To test for differences in response rates both following the initial email requesting 
survey participation, as well as at the end of survey administration, a series of chi-square 
tests were conducted. As seen in Table 6, only one significant finding emerged: at the 
conclusion of the survey, the response rate for those offered the $100 incentive (16.2%) 
was significantly greater than the response rate for the control group (13.9%), % 2 (1) = 
3.93, p = 0.047. This finding may imply that the relationship between incentive amount 
and survey response is non-linear, as it is clear from the response rates and the statistical 
tests that “more” is not better: response rates did not increase as the amount of the 
incentive increased. However, given our large sample size combined with the marginal 
p-value, as well as the weak substantive impact of a 2.3% increase in response rate, the 
strength of support these data give for a non-linear effect may be suspect. 

In addition to examining the effect of incentives on response rates, we also tested 
whether our experimental conditions had any effect on the quality of survey response. It 
is possible that an incentive might not change the probability that an individual will 
respond to a survey, but it might cause respondents to spend more time answering the 
survey. One way to test this hypothesis is to test item non-response between the 
experimental groups. If this is happening, we would expect lower item non-response for 
respondents in the incentive groups. 

Our survey was adapted from the College Board’s Admitted Student Questionnaire 
Plus and was comprised of six topics: 

• Importance of college characteristics 

• Characteristics ratings for the university 

• Role of financial aid in the application process 

• Images of the university; 

• Number of applications mailed 

• Demographic information 

For each respondent, we calculated the number of survey items completed in each of the 
first five sections separately and these scores served as, dependent measures in a series of 
one-way ANOVAs. For the demographic variables, we recorded whether or not 
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respondents supplied the requested information, and used the resultant binary data (0= did 
not supply; 1= supplied) in a series of chi-square tests. 

In the series of one-way ANOVAs, we examined if (1) the number of survey items 
completed, or (2) the mean responses given varied across the survey conditions. As in 
earlier analyses, we tested the three main research questions outlined above. 

Table 7 shows the results of the analyses conducted using the number of survey 
items completed or response/non-response as dependent measures. Significant effects of 
lottery incentives were only found for the importance of college characteristics and the 
number of college applications. We found no significant findings for the number of items 
completed for: the importance of characteristics, financial aid, or images of the 
institution, or the provision of demographic information. 

With the exception of the $150 incentive, the mean number of items completed by 
each level of incentive was found to differ from the control group in the characteristics 
ratings section of the survey. Specifically, respondents in the $50, $100, and $200 
incentive groups completed more items than respondents in the control group, with means 
of 16.8, 16.8, 16.9, and 16.3 (out of 17 items), respectively. Additionally, a significant 
effect of the overall impact of offering incentives was found for the number of 
characteristics ratings completed, with respondents offered incentives completing 
significantly more items ( m = 16.75) than the control (m = 16.29). These findings suggest 
that the use of incentives may have caused respondents to complete a greater number of 
items specific to the university offering the reward, while more general survey questions 
were completed at a rate that was identical to respondents not offered an incentive. 
However, the substantive difference is quite small: about .5 items. 

Analysis of the number of college applications found that respondents in the $200 
incentive group applied to significantly more schools ( m = 5.6) than the control ( m = 5.0). 
This finding can be interpreted two ways. The first interpretation posits that the 
possibility of a large reward caused respondents in the largest incentive category to 
complete the survey more thoroughly. The second, and more plausible interpretation is 
that this finding is simply spurious - if the first interpretation were true, then we would 
have expected the analyses to reveal a larger number of significant findings. 

To compare opinions of the survey groups on the importance of college 
characteristics in the application process and the characteristic rating section of the 
survey, we conducted a series of one-way ANOVAs using the mean response to each item 
as the dependent measure. Of the 204 tests conducted (34 survey items times 6 
comparisons), only 6 (2.9%) were found to be significant. Because the number of 
significant effects was about what we might expect to find erroneously (at p < 0.05), we 
concluded that these (significant) findings were spurious. 

In sum, the offer of a $100 gift certificate in a drawing increased the response rate 
by 2 percentage points, but there were no other significant differences between the control 
and incentive groups. Given the very large sample size and p value, this is a weak 
finding, especially in terms of the substantive effect. In addition, it does not appear that 
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offering larger amounts of incentives has a positive impact on response. There was some 
evidence that members of the incentive groups spent more time on the survey, as 
indicated by a slightly smaller item non-response rate. Again, the substantive differences 
were small. 



Limitations 

The chief limitation to our study is that the survey population is still not a college 
student population. It is possible that students’ receptiveness to lottery incentives may 
change from their senior year in high school to when they enter college. 

The overall response rate for the survey may also pose a problem. With such a 
low response rate, one could argue that interest in the survey was apparently so low that 
no incentive could have made a substantively large impact on response rates. 
Alternatively, the opposite argument could also be made. It is in low-interest surveys 
where incentives should make a difference and be effective, as respondents have few 
other reasons to participate. If respondents are very interested in a survey because of its 
content and thus are likely to respond, incentives may not have much of an impact beyond 
this interest. 

It is also possible that given amount of the amount of spam and unwanted 
solicitation that are sent via email, we might have had more success with a paper survey. 
With a paper survey the lottery offer might have been more believable. We believe this 
had a minimal impact on our study, as our emails contained “institution.edu,” sending a 
signal that we were members of a higher education institution (and thus increasing our 
credibility). The survey was also clearly located in the university domain. 

The impact of the odds of winning a lottery is a further limiting factor for the 
study. In this study respondents could not estimate the odds of winning, as they had no 
idea how many other people had received an invitation to participate. For typical student 
survey respondents would have a rough idea of their odds of winning given the size of the 
student body and the number of prizes offered. Thus a lottery incentive might have more 
of an effect, as respondents would be better able to estimate the expected value of the 
incentive. Little, if any, research has been conducted on the impact of odds on lottery 
incentives and response rates, so this can only be a speculation as to what might have 
occurred if we had been able to conduct our analysis on a college student population. 

Conclusion 

Although the literature on incentives and response rates shows that post-payment 
of incentives in general and lotteries in particular have little or no impact on survey 
response, use of such lottery incentives appears common in institutional research. Our 
research is in line with previous research on the minimal effect of post-paid incentives. 
This raises a serious question of effectiveness and resource allocation. Given limited 
resources, should we be spending time and money on awarding prizes, or on efforts 
proven to increase response rates, such as Dillman’s (2000) method? 




- 128 - 

130 



References 



La Mar Adams, L. & Gale, D. (1982). Solving the quandary between questionnaire length and 
response rate in educational research. Research in Higher Education, 17 (3), 231-240. 

Boser, J. (1990). Surveying alumni by mail: effect of booklet/folder questionnaire format and 
style of type on response rate. Research in Higher Education, 31 (2): 149-159. 

Church, A. (1993). Estimating the effect of incentives on mail survey response rates: a meta- 
analysis. Public Opnion Quarterly, 57 : 62-7. 

Cote, L., Grinnell, R. and L. Tompkins (1986). Increasing response rates to mail surveys: the 
impact of adherence to Dillman-like procedures and techniques. Review of Higher 
Education, 9 (2): 229-242. 

Couper, M. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 
64; 464-494. 

Dey, E. (1997). Working with low survey response rates: the efficacy of weighting adjustments. 
Research in Higher Education, 38 (2): 215-227. 

Dillman, D. (2000). Mail and Internet Surveys: The Tailored Desien Method . New York: John 
Wiley & Sons. 

Grosset, J. (1995). The biasing effects of bad addresses on information gathered by mail surveys. 
Journal of Applied Research in the Community College, (2) 2: 179-191. 

Groves, R., Cialdidn, R., and Couper, M. (1992). Understanding the decision to participate in a 
survey. Public Opinion Quarterly, 56 : 475-495. 

James, J., & Bolstein, R. (1992). Large monetary incentives and their effect on mail survey 
response rates. Public Opinion Quarterly, 56 . 442-453. 

Schiltz, M. (1988). Professional standards for survey research. Research in Higher Education, 
(28)1: 67-75. 

Singer, E., Hoewyk, J. V., & Maher, M. P. (1998). Does the payment of incentives create 
expectation of effects. Public Opinion Quarterly, 62 . 152-164. 

Singer, E., van Hoewyk, J., & Maher, M. (2000). Experiments with incentives in telephone 
surveys. Public Opinion Quarterly, 64 , 171-188. 

Singer, E., Thum, D. R. V., & Miller, E. R. (1999). Differential incentives: Beliefs about 
practices, perceptions of equity, and effects on survey participation. Public Opinion 
Quarterly, 63 . 251-260. 




- 129- 



131 



Smith, K. and T. Bers (1987). Improving alumni survey response rates: an experiment and cost- 
benefit analysis. Research in Higher Education , 27(3): 218-225. 

Smith, T. (1995). Trends in nonresponse rates. International Journal of Public Opinion Research, 
7,157-171. 

Warriner, K., Goyder, J., Gjertsen, H., Hohner, P., & McSpurren, K. (1996). Charities, no; 
Lotteries, no; Cash, yes: Main effects and interactions in a Canadian incentives 
experiment. Public Opinion Quarterly, 60 . 542-562. 

Willimack, D., Schuman, H., Pennell, B., & Lepkowski, J. (1995). Effects of a prepaid 

nonmonetary incentive on response rates and response quality in a face-to-face survey. 
Public Opinion Quarterly, 59 . 78-92. 



Table 1. Number of Surveys Using a Lottery Incentive During a Typical Academic Year 



Number of surveys 
with prizes 


All schools 


Private 


Public 


N 


% 


N 


% 


N 


% 


0 


222 


64% 


53 


50% 


164 


69% 


1 


75 


22% 


33 


31% 


41 


17% 


2 


31 


9% 


15 


14% 


16 


7% 


3 


11 


3% 


2 


2% 


9 


4% 


4 


5 


1% 


1 


1% 


4 


2% 


5 + 


3 


1% 


1 


1% 


2 


1% 


Total 


347 


100% 


105 


100% 


236 


100% 



Note: all schools contain respondents who did not identify their school? 
Table 2. Types of Prizes Used 



Prize 


Total 


Private 


Public 


Cash 


22.1% 


21.6% 


22.9% 


Gift certificate - national 


11.5% 


21.6% 


2.9% 


Gift certificate - local 


23.0% 


29.4% 


17.1% 


Gift certificate - school 


57.4% 


.62.8% 


54.3% 


Travel prize 


5.7% 


11.8% 


1.4% 


Electronics 


5.7% 


7.8% 


4.3% 


Clothing 


14.8% 


13.7% 


15.7% 


Other type of prize 


25.4% 


17.7% 


31.4% 



Table 3. Perceived Impact of Lottery Incentives on Response Rates 



Researchers Researchers 
All not using using 

respondents incentives incentives 





N 


% 


N 


% 


N 


% 


Greatly decrease response rates 


0 


0.0% 


0 


0.0% 


0 


0.0% 


Slightly decrease response rates 


1 


0.5% 


0 


0.0% 


1 


0.8% 


No effect at all on response rates 


50 


24.2% 


39 


46.4% 


11 


8.9% 


Somewhat increase response rates 


141 


68.1% 


41 


48.8% 


100 


81.3% 


Greatly increase response rates 


15 


7.3% 


4 


4.8% 


10 


8.9% 


Total 


207 


100% 


84 


100% 


122 


100% 
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Table 4. Survey Experiment Groups 



Group 


N of 

High schools 


Students 


Mean number of 
students per school 


Control (no incentive) 


706 


1,983 


2.9 


Incentive - $50 


706 


1,712 


2.5 


Incentive - $100 


706 


1,960 


2.8 


Incentive - $150 


706 


1,784 


2.6 


Incentive - $200 


705 


1,866 


2.7 


Total 


3,529 


9,305 


2.7 



Table 5. Initial and Final Response Rates by Incentive Group 



Group 


After 1st email 


After 3rd email 


Control (no incentive) 


4.6% 


13.9% 


Incentive - $50 


5.4% 


15.0% 


Incentive - $100 


5.3% 


16.2% 


Incentive - $150 


6.0% 


15.6% 


Incentive - $200 


5.8% 


15.4% 


All incentive groups 


5.6% 


15.6% 


Total sample 


5.4% 


15.2% 



Table 6. Hypotheses and Tests for Differences in Response Rates 



Dependent variable: response rate 





After 1st email 


After 3rd email 


Hypothesis 


Z 2 


df 


P< 


z 2 


dj 


P< 


Control 4 $50 4 $100 4 $150 4 $200 


4.27 


4 


0.371 


4.30 


4 


0.367 


Control 4 $50 


1.26 


1 


0.262 


0.97 


1 


0.324 


Control ^ $100 


1.03 


1 


0.310 


3.93 


1 


0.047 


Control ^ $150 


3.45 


1 


0.063 


2.15 


1 


0.142 


Control 4 $200 


2.97 


1 


0.085 


1.74 


1 


0.187 


Control 4 all incentive groups combined 


3.17 


1 


0.075 


3.36 


1 


0.067 



Table 7. Hypotheses and Tests for Differences in Item Non-Response 
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Introduction 

The Internet is quickly becoming an integral part of everyday life. Since web- 
based surveys may be beneficial in terms of cost, time, and accuracy, more and more 
survey research (including that done in higher education and Institutional Research) is 
being conducted on the web (Underwood, Kim, & Matier, 2000). Thus, it is important to 
determine whether or not data obtained from web surveys are as valid and reliable as that 
of paper surveys, if response rates differ between paper and web surveys, and how ethical 
issues may effect the data collection, results, and reporting of the survey data. Moreover, 
ethical issues such as anonymity and/or confidentiality, the representativeness of the data, 
data analysis, and the use of quotations will need to be addressed when administering 
surveys on the web (Goree & Marszalek, 1995). 

Only a handful of studies have explored how respondents react to web surveys, 
looking at issues such as bias in responses compared to paper and pencil surveys, or 
differences in overall response rates. With respect to response rates, Underwood et al. 
(2000) found that one mail (paper) survey had a notably higher response rate than another 
survey administered on the web. In addition, some research has indicated that females 
tend to respond in a greater proportion than males regardless of the method of survey 
administration (Underwood et al., 2000), while other results suggest that web surveys 
yield a greater proportion of male than female respondents (Tomsic, Hendel, & Matross, 
2000). Moreover, recent findings suggested that certain ethnic minorities tend to respond 
at lower rates than do White, Asian American, and International Students regardless of 
the survey administration method (Underwood et al., 2000). 

Other researchers have explored the issue of response bias or differences in 
responses based on method of survey administration. Baron and Siepmann (2000) found 
that responses to web surveys were very similar to the responses to paper surveys, but 
only if the formatting was exactly the same. Further, some studies have found that web 
surveys tended to receive significantly more favorable responses than did paper surveys 
(Carini, Hayek, Ouimet, & Kuh, 2001; Tomsic et al., 2000). Research conducted by 
Underwood et al. (2000) suggested that web surveys elicited less favorable responses than 
did paper surveys. Some findings have indicated that respondents tend to respond in a 
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